Nowadays, fine-grained Human Activity Recognition (HAR) has become extremely interesting among researchers due to its applications in fields such as healthcare, security, sports, and smart environments. In this paper, we provide a brief overview of the State of the Art approaches in fine-grained human activity recognition. We also discuss the characteristics, complexities, and scarcity of inertial datasets related to fine-grained and coarse-grained activities. To mitigate this scarcity, we collect our inertial dataset, consisting of 17 participants performing 4 fine-grained tasks while interacting with an Inertial Measurement Unit (IMU) sensor embedded in a solid object. Next, we test the most commonly used machine learning classifiers (e.g., kNN, XGboost) on the collected dataset and present the results. Finally, we demonstrate the necessity of a new approach to deal with the recognition of fine-grained activities, and we state our future research directions in this context.