Cryptocurrencies have won a lot of attention as an investment tool in recent years. Specific research has been done on cryptocurrencies’ price prediction while the prices surge up. Classic models and recurrent neural networks are applied for the time series forecast. However, there remains limited research on how the Transformer works on forecasting cryptocurrencies price data. This paper investigated the forecasting capability of the Transformer model on Bitcoin (BTC) price data and Ethereum (ETH) price data which are time series with high fluctuation. Long short term memory model (LSTM) is employed for performance comparison. The result shows that LSTM performs better than Transformer both on BTC and ETH price prediction. Furthermore, in this paper, we also investigated if sentiment analysis can help improve the model’s performance in forecasting future prices. Twitter data and Valence Aware Dictionary and sEntiment Reasoner (VADER) is used for getting sentiment scores. The result shows that the sentiment analysis improves the Transformer model’s performance on BTC price but not ETH price. For the LSTM model, the sentiment analysis does not help with prediction results. Finally, this paper also shows that transfer learning can help on improving the Transformer’s prediction ability on ETH price data.