A mixed graph is a set of vertices together with an edge set and an arc set. An $(m,n)$-mixed graph $G$ is a mixed graph whose edges are each assigned one of $m$ colours, and whose arcs are each assigned one of $n$ colours. A \emph{switch} at a vertex $v$ of $G$ permutes the edge colours, the arc colours, and the arc directions of edges and arcs incident with $v$. The group of all allowed switches is $\Gamma$. Let $k \geq 1$ be a fixed integer and $\Gamma$ a fixed permutation group. We consider the problem that takes as input an $(m,n)$-mixed graph $G$ and asks if there a sequence of switches at vertices of $G$ with respect to $\Gamma$ so that the resulting $(m,n)$-mixed graph admits a homomorphism to an $(m,n)$-mixed graph on $k$ vertices. Our main result establishes this problem can be solved in polynomial time for $k \leq 2$, and is NP-hard for $k \geq 3$. This provides a step towards a general dichotomy theorem for the $\Gamma$-switchable homomorphism decision problem.