In recent years, reinforcement learning algorithms have become very popular in solving robotic manipulation tasks. However, these algorithms usually suffer from sample inefficiency, meaning that they require extensive exploration to find an appropriate control policy capable of solving the robotic manipulation tasks targeted, and, consequently, this sample inefficiency leads to training time inefficiency. The work presented in this thesis aims to solve this problem through three main methods: (i) reward engineering; (ii) a sequential execution of primitive skills; and (iii) an automatic execution of primitive policies. In particular, a novel reward that encourages lifting behaviour is presented and evaluated in the context of the Real Robot Challenge competition problem. A novel learning framework for in-hand object 3D rotations based on sequentially executed primitives is proposed and evaluated in the Shadow Dexterous Hand simulated environment of the OpenAI Gym. Finally, a new algorithm is proposed that is capable of automatically deciding, during exploration, whether or not to use a primitive behaviour, with this approach evaluated in different tasks for different manipulators available in the OpenAI Gym.