hi, I am searching for a software or library that can handle operator
algebra in quantum mechanics. For example, I would like to caculate
the commutator [a,H], where the Hamiltonian "H"=\epsilon a^{\dag}a,
and "a" is annihilation operator for, say, an osilator. I tried
Maxima, and GiNaC, and read many documentation of them, and get little
clue. I wonder if there is direct implementation of these kind of
operator algebra in any software or library.

Thank you in advance.
Sure. Try Form, it is specialised in those kind of operations.