In this paper, a cryptosystem based on chaos and Brahmagupta and Bhaskara (BB) equation and its VLSI architecture are proposed. In a chaotic binary sequence, the gray level of each pixel is XORed or XNORed bit-by-bit to one of the two predetermined keys. This new scheme is used to modify the pixel values with pseudorandom sequences generated by linear feedback shift register (LFSR). Based on the characteristics of the pseudo-random, unpredictability of chaotic sequences and extreme sensitivity to the initial conditions are very high. The simulation results show that the image encrypted by this new scheme can not be recognized. Better mean square error and peak signal to noise ratio of the images can be obtained. Its features are low computational complexity, high security and no distortion. In order to implement the algorithm, VLSI architecture with low hardware cost, high computing speed, and high hardware utilization efficiency is also proposed.