Last year, Apple deployed a mechanism on its devices that identifies landmarks in user photos and then allows them to find pictures on users’ mobile devices and computers based on the names of those same landmarks. The manufacturer enabled this feature without the explicit consent of device owners, and they have only now begun to notice this.
Image source: apple.com
The feature was called “Enhanced Visual Search,” and was first announced to the public by programmer Jeff Johnson. According to him, the new service began working on October 28, 2024 on iOS 18.1 and macOS 15.1, and Apple did not bother to clearly explain how this technology works.
A document dated November 6, 2024 on Apple’s website states: “Improved visual search in the Photos app lets you search for photos based on landmarks and landmarks. Your device confidentially matches the locations in your photos with a catalog of global locations stored on Apple servers. We use homomorphic encryption and differential privacy technologies, and also use an OHTTP protocol node that hides the IP address. This prevents Apple from collecting any information about what is shown in your photos. You can turn off Enhanced Visual Search at any time. To do this, on an iOS or iPadOS device, open “Settings” –> “Applications” –> “Photos”. On a Mac, open the Photos app and select Settings -> General..
A detailed explanation of how the technology works is provided in an article on Apple’s website dated October 24, 2024, around the same time that Enhanced Visual Search was launched. It is based on a machine learning model that runs locally on devices. She analyzes photographs for the presence of landmarks. If the AI finds a match, the system calculates a vector embedding—an array of numbers representing that portion of the image. The vector embedding is then encrypted using homomorphic encryption, which allows computations to be performed on the encrypted data without decrypting it.
After this, the data is sent to a remote server, where a search is performed in the database of attractions. The results are returned to the user’s device, which Apple claims is the only one capable of decrypting the information. An additional level of protection is the differential confidentiality mechanism, which excludes the identification of a subject in the database, even if an attacker gains access to the entire database.
However, experts interviewed by the British publication *The Register* expressed doubts about the new Apple service. The company has enabled “Enhanced Visual Search” by default for all users, including those who have turned off photo uploads to iCloud for privacy reasons.
More than three years ago, Apple planned to scan images on users’ devices for child abuse material, but abandoned the initiative due to privacy concerns. The new system, critics say, forcibly processes locally stored photos and sends data about all images, not just those with suspicious hashes.
Additionally, it remains unclear why Apple launched “Enhanced Visual Search” without making sure users of its devices were properly notified.
If you notice an error, select it with the mouse and press CTRL+ENTER.
