Privacy-preserving neural networks learning

From CryptoWiki
Jump to: navigation, search



Applying machine learning to problems that include medical, financial, or other types of sensitive data requires not only accurate predictions, but careful attention to maintaining data privacy and security. Legal and ethical requirements prevent the use of cloud-based machine learning for this kind of task [1]. This article presents a method for converting trained neural networks that can be applied to encrypted data. This allows the data owner to send their data in encrypted form to a [cloud] service that contains a neural network.Encryption ensures that the data remains confidential because the cloud does not have access to the keys needed to decrypt. Nevertheless, the cloud service is able to apply the neural network to the encrypted data, to generate encrypted forecasts, and also to return them in encrypted form to the copyright holder. These encrypted predictions can be sent back to the owner of the secret key that can decrypt them.herefore, the cloud service does not receive any information about the raw data or about the forecast made by it. Consider a hospital that would like to use the cloud service to predict the likelihood of a patient being hospitalized again over the next 30 days in order to improve the quality of care and reduce costs. Due to the ethical and legal requirements regarding the confidentiality of patient information, the hospital may be prohibited from using such a service. This article presents the way in which a hospital can use this service without sacrificing patient confidentiality.In the proposed protocol, the hospital encrypts personal information and sends it in encrypted form to the cloud. The cloud is able to generate a forecast for the encrypted data records and send back the results that the hospital can decipher and read. The encryption scheme uses the public key for encryption and the private key (private key) for decryption. It is important to note that the cloud does not have access to the private key, so it cannot decrypt the data and cannot decrypt the prediction. The only information he receives during this process is that he performed the forecast on behalf of the hospital.When using a homomorphic encryption scheme, you can encrypt your data in advance before entering it into the machine learning model. This procedure will allow you to make private and secure forecasts without the need to establish trust between the data owner and the service provider. This can be applied in areas such as healthcare, finance, business, etc. One of the encryption schemes, which has a homomorphic property, is the Paillier encryption scheme, first introduced in 1999 by Pascal Paillier. This encryption scheme has proven its semantic security against plaintext attacks.Paillier encryption scheme supports the addition of ciphertexts and the multiplication of ciphertext by plain text [2]. With this property, we can do matrix multiplications and vector additions. Similar to the fact that the trained model of a small neural network can be simplified to matrix multiplication and vector addition (The principle of the neural network in cryptography can be found by clicking on the link ), therefore, this model can accept input data encrypted using an encryption scheme.

Applying Paillier Encryption Scheme

Figure 1 - Paillier encryption scheme.

Paillier encryption scheme is an asymmetric encryption scheme. It uses Perekosnk.jpg as the public-key,where Perekosn.jpg - 𝑛 is a product of two big prime numbers Perekosp.jpg and Perekosq1.jpg, and Perekosq.jpg is an element of Perekosz2.jpg such that its order is a multiple of Perekosn.jpg. The two prime numbers Perekosp.jpg and Perekosq1.jpg are used as a secret-key. One version of this encryption scheme sets Perekosq.jpg equals to Perekosn+1.jpg and requires Perekosnk.jpg to be a prime of equal length (in their binary representation). The requirement assures that Perekosfi.jpg has an inverse modulo Perekosn.jpg. The inverse of Perekosfn.jpgis denoted by Perekosm.jpg and used as the secret-key [5]. The scheme is summarized in [Figure 1] . Suppose that two plaintexts Perekosm1.jpg and Perekosm2.jpg are encrypted toPerekos1-1.jpg and Perekos1-2-1.jpg where Perekosr1r2.jpg. We can see that Perekos1-4.jpg Perekos1-5.jpg

Since Perekosq.jpg has order Perekosn.jpg in Perekosz2.jpg and Perekos1-3.jpg then Perekos1-6.jpg

Furthermore, if Perekosm1m2n.jpg, then


Because of this property, Paillier encryption scheme is called a partially homomorphic encryption scheme. This property can be generalize to addition of Perekosn.jpg plaintexts 3-1-1.jpg

Now suppose that Perekosk.jpg is a positive integer then


Since Perekosq.jpg has order Perekosn.jpg in Zn.jpg and Perekosr3.jpg


Furthermore, if Perekoskm.jpg then


We will make use of these two properties to build a system where a machine learning model can take an input encrypted by Paillier encryption scheme and produce an output encrypted with the same key.

The Machine Learning Model

First, we will train a neural network model with description given as below:

1.Input layer with z1 neurons corresponding to flattened array of the MNIST [4] image data.

2.First hidden layer: fully connected (dense) layer with z2 neurons and identity function as the activation function.

3.Second hidden layer: fully connected layer with z3 neurons, also having identity function as the activation function.

4.Output layer with z4 neurons corresponding to z4 classes of the MNIST database .

5.There are (z1 × z2) + (z2 × z3) + (z3 × z4) weights parameter and z2 + z3 + z4 biases parameter that can be optimized in the training process. This model can be written in a form of matrix multiplications as below:

4-1.jpg (3)



  • 4-3.jpg and 4-4.jpg respectively is a column vector of length z1, z2, z3 and z4.
  • 4-5.jpg and 4-6.jpg respectively is a column vector of length z2, z3, z4.
  • 4-7.jpg and4-8.jpg respectively is a matrix of size z2 х z1, z3 х z2, z4 х z3.
  • 4-9.jpg or 4-4.jpg is the output of the model in the form of a column vector of length z4.

After the training phase, the weights and biases will have a fixed value, so the model can then be simplified as .




is a matrix of size z4 х z1 and a column vector of length z4 respectively [6]. Because of the softmax activation function in the output layer, this model cannot take an input encrypted with Paillier encryption scheme yet, softmax function is not a function that only consists of addition and scalar multiplication. However, we will see that the model is able to predict the input correctly without computing the softmax function in the output layer. Neural networks model determine the inputs’ class by looking at the index of the maximum value of 4-9.jpg.On the other hand, softmax function dosen’t change the order of its input, for example if the input vector is 4-13.jpg и 4-15.jpg we see that the elements of. In general, suppose that 4-14.jpg and we can assume that 4-16.jpg since 4-17.jpg is greater than 1 we can see that4-18.jpg4-19.jpg finally since 4-20.jpg is positive then 4-21.jpg тthus we have proven that in general case the elements of 4-13.jpg and 4-15.jpg have the same order. We can now safely drop the softmax function in the output layer of the model. The model is now simplified to

4-22.jpg (4)

Suppose that Alice have successfully trained a model that has a structure like [(3)] which she then simplify to [(4)]. She then builds a web application so that people can access it easily via internet. Bob wants to use the model with his private data 4-23.jpg. The output that Bob looking for 4-24.jpg. Because 4-30.jpgis private, Bob will encrypt it first to 4-25.jpg which he then inputs it to Alices’ app. In order to get 4-26.jpg, the app have to compute

Figure 2 - Summary.

4-27.jpg (5)

which equals to


Therefore, using equation [(5)] Alices’ app can outputs 4-26.jpg, when given 4-29.jpg as input. Bob can decrypt this this output to 4-9.jpg,the value that he was looking for from the trained model. This way, Bob can still obtain the value that he was looking for from Alices’ model without having to give his private data to Alice. Furthermore, Alice, as the application admin, won’t be able to read the output— which may be confidential as well—since it is in an encrypted form. [Figure 2] gives an illustration on how this concept works


Paillier encryption scheme is homomorphic with addition and scalar multiplication of ciphertexts. Using the weight and dimensions of the simplified model, you can form an encrypted prediction for sensitive data.



Go to Bibliography"Editing Privacy-preserving neural networks learning"

Perekos.V.A. Moscow, 2019