Generating 3D Models in Mobile: Sony’s 3D Creator Made Me a Bobblehead

In a show like IFA, it’s easy to get wide-eyed about a flashy new feature that is being heavily promoted but might have limited use. Normally, something like Sony’s 3D Creator app would fall under this umbrella – a tool that can create a 3D wireframe model of someone’s head and shoulders and then implement a 4K texture over the top. What is making me write about it is some of the implementation.

Normally in a single photo, without subsequent depth map data, creating a 3D model is difficult. Also, depth data would only show points directly in front of the camera – it says nothing about what is around the corner, especially when it comes to generating a texture from the image data to fit the model. With multiple photos, by correlating points (and perhaps using internal x/y/z sensor data), distances can be measured for identical points and a full depth map can be done taking the color data from the pixels and understanding which pixel would be where in that depth map allows the wireframe model to be textured.

For anyone who follows our desktop CPU coverage, we’ve actually been running a benchmark that does this for the last few years. Our test suite runs Agisoft Photoscan, which takes a set of high-quality images (usually 50+ images) of people, of items, of buildings and of landscapes, and builds a 3D textured model to be used in displays, games, and anything that wants a 3D model. Normally this benchmark is computationally expensive: Agisoft splits the work into four segments:

1. Alignment
2. Point Cloud Generation
3. Mesh Building
4. Texture Building/Skinning

Each of these segments has dedicated algorithms and the goal here is to compute as fast as possible. Some of the algorithms are linear and rely heavily on single thread performance, whereas others, such as Mesh Building, are very parallel which Agisoft implements via OpenCL. This allows any OpenCL connected accelerator, such as a GPU, to be able to increase the performance of this test. For low core count CPUs this is usually the longest part of the full benchmark, however the higher core count parts move into other bottlenecks, such as memory or cache.

So for our Agisoft run in those benchmarks, we use a 50 image set of a building with 4K images. We get the algorithm to select 50000 points from each image, and use those for the mesh building. We typically run it in OpenCL off mode, as we are testing the CPU cores, although Ganesh has seen some minor speedup on this test with Intel’s dual-core U-series CPUs when enabling OpenCL. A high end but low power processor, such as the Core i5-7500T, takes nearly 1500 seconds, or 25 minutes to run our test. We also see speed up based on cache sizes and DRAM frequency/latency, but major parts of the app either rely on single thread performance exclusively or multithread performance exclusively.

Sony’s way of creating the 3D head model involves panning the camera from one ear to the other, and then moving the camera around the head to generate finer detail and texture information. It does this all in-situ, computing on the fly and showing the results in real time back on the screen as the scan is being done. The whole process takes a minute, which compared to the method outlined above, is super quick. Now of course, Sony’s implementation is limited to just heads, rather than something about buildings, and we were told by Sony that their models are limited to 50000 polygons. During the demonstration I was given, I could see the software generating points on the head and it was obvious the number of points was in the hundreds in total, rather than the thousands per static image, so there is a perceptible difference in quality. But the Sony modeling implementation still gives a good visual output. 

The smartphones from Sony that support this feature are the XZ series, which have Snapdragon 835 SoCs inside. Qualcomm is notoriously secretive about what is under the hood on their mobile chips, although features like the Hexagon DSP contained within the chip are announced. Sony would not state how they are implementing their algorithms, if they were leveraging a compute API from the Adreno GPU, a graphics API, the Kryo CPUs, or something from the special DSPs housed on the chip. It also leads two different questions: do the algorithms work on other SoCs, or can other Snapdragon 835 smartphone vendors develop their own equivalent application?

Sony’s goal is to allow users to implement their new facial model in applications that support personal avatars, or exporting to 3D printing formats for real-world creation of a user’s head. My mind instantly pointed to who would use something like this on scale: console players, specifically on the Xbox and Nintendo devices, or for special games such as NBA2k17. Given Sony’s exists in the console space with their own Playstation 4, one might expect them not to play with competitors, although the smartphone department is a different business unit (and other Snapdragon 835 players do not have a potential conflict). I was told by the booth demonstrator that he doesn’t know of any collaboration, which is unfortunate as I’d suspect this as being a good opening for this tool.

I’m trying to probe for more information, from Sony on the algorithm or Qualcomm on the hardware, because how the algorithm is implemented on the hardware is something I find interesting given how we’ve tested desktop CPUs in the past. It also puts the challenge to other smartphone vendors that use Snapdragon 835 (or other SoCs) to see if this is a feature that they might want to implement, or if there are apps that will implement this feature regardless of hardware.

Gallery: Generating 3D Models in Mobile: Sony’s 3D Creator Made Me a Bobblehead

Related Reading

• Sony Launches Xperia XZ Premium and Xperia XZs Phones For US Market
• Sony Announces Xperia XZ and Xperia X Compact
• The Sony Xperia X Preview

Hands on with the LG V30/V30+: 6-inch OLED 2880×1440 Display, Quad DAC, IP68, Daydream VR

As part of their IFA 2017 showcase, this morning LG is unveiling their long-rumored next-generation flagship phone, the V30. Like its predecessor, the LG V20, the V30 hasn’t exactly been a well-kept secret, with several leaks springing over the last month. Still, with the official announcement from LG, we’re finally getting a complete look at the phone.

Some of the feedback that LG received for the LG V10 and the LG V20 explained that the smartphones were considered ‘too professional’ for the mass market and younger generations. Despite LG’s heavy marketing to millennials using celebrities such as Joseph Gordon-Levitt, the dual screen combinations had an interesting feature set but often sat in the store next to their better selling competitors. To crank up the base of Generation V (what LG calls the millennial generation targeted by the V-series), the biggest and most noticable design change for the V series is that the new V30 drops the slightly esoteric dual screen design in favor of a single, larger FullVision OLED display, with super narrow bezels to make as much of the phone a display as possible.

For our hands-on, LG requested that we clarify that the devices are pre-production preview models, and some features/issues may change between now and launch.

Under the hood, the V30 is powered by Qualcomm’s Snapdragon 835, with LG using a heatpipe to assist in cooling. This is paired with 4 GB of LPDDR4X, and either 64GB or 128GB of UFS 2.1 storage. The 128GB model will be sold as the V30+ model, and will only be available in some countries (I was told that the V30 itself is not fully worldwide, and the V30+ will still cover most of those but not all). The battery is virtually unchanged from the V20, its 3300 mAh capacity being just 100 mAh larger than its predecessor. Some journalists suggested that this might be a design oversight, given the larger size and OLED display in this unit – LG stated that the battery life is still good for an honest day of use. On a personal note, just be wary of the brightness: LG does offer screen dimension reduction tools, which might help here. With the battery the V30 is supporting Qualcomm’s Quick Charge 3.0 technology, giving 50% charge in 30 minutes under supported chargers, and the V30 will also have wireless charging.

Overall, the main display is the hardest thing to miss when seeing the phone for the first time. The 6-inch diagonal QHD+ (2880×1440) resolution display is big and bright. And even though I’ve used phones like the Mi MIX, which does things like crazy placement, for all intents and purposes this is a very narrow bezel phone, and a large phone at that. LG uses their pOLED technology here, which they were keen to point out is still an ‘active matrix’ type of OLED, however AMOLED is the Samsung brand name. The technology means that LG is supporting HDR10 out of the box for suitable content, boasts an always-on lock screen at super low power, and can crank up the brightness that’ll tear through your battery life in a few short hours. With the 6-inch display, OLED, and a high 537 pixels per inch, LG is also supporting Daydream VR with the new V30.

The aim for the V30 is all about visuals. A better display, a better camera, new camera features, and overall a new visual experience for users that share content either as a video creator or as an image junkie. Moving to the camera, LG is employing a dual camera system, using a 71° degree main camera with the new Sony IMX351 sensor supporting OIS, and a 120° wide angle camera for wider shots and some zoom functionality. The primary rear camera is a 16MP design with a f/1.6 aperture, noted as being the widest in the market. This is combined with a new lens strategy, whereby the 6-element lens now makes one of them glass (rather than them all being plastic), which LG states as offering 4% more light into the sensor. We were told that the first lens element is the one updated, as when they tested the full range, this made the most difference. On the back of this change, LG is stating that their dynamic range is wider than their competitors, offering 11 stops at 6 dB rather than 9.5, and competing with more expensive cinema cameras and beating the smartphone competition. This also means wide color gamut support, using logarithmic color imaging (rather than a linear scale) for a much wider dynamic range. The log color support is via a new feature for their camera, LG Cine Log.

LG spent a lot of time talking to us about Cine Log. Using the manual video mode and enabling Cine Log, and recording the color information with the MPEG4 video, users can combine the data with their own custom look-up-tables using software such as DaVinci Resolve in order to adjust the color tone mapping. Inside the Cine Log portion of the app, users can already select 15 different cinematic effects that LG used experienced outside video editing professionals to help calibrate. This includes effects such as romantic, drama, mystery, thriller, documentary, noir, flashback, scenery, melodrama, classic and pop-art. On the face of it, these might appear to be extensions of popular filter apps, but LG states that their solution is more than just a simple filter, using specific color enhancement. We took a few photos of a lamp in a dimly lit room and could see a difference.

  
Documentary, Blockbuster, Beauty

Another feature of Cine Log is point-zoom, which allows the user to zoom in to any part of an image, rather than requiring the camera to be moved to actually face the object in the dead center.

Not specifically in Cine Log is an app called Graphy, providing a series of stock photos with various ‘looks’ and feels to them. Users can recreate the professional settings of these photos for any images of their own by selecting them – the app applies all the required settings.

For video recording, LG is promoting the inclusion of two high-performance microphones in the design. With the addition of the standard phone receiver, the internal DSP will combine the audio from all sources to provide a single output (although they don’t state what the target SNR of the path is). LG stated that this audio is good at 24-bits and 96 kHz, though that doesn’t state anything about the quality. I played around with the feature, and it does offer some immediate features that should be on all smartphones: additional gain jumps, low-frequency filters, and a limiter for clipping.

It’s worth noting at this point in our briefing, one of the analysts mentioned that at the beginning LG was targeting ‘Generation V’ / millennials with their new phone, but so far all the features being presented were still very hard technology tools for experts and professionals. LG responded by saying that most of these features were available as one-button behind the scene applications: many of these users know how to apply a filter, and the application of their tools that are more complicated underneath the surface can be done in the same way.

LG was keen to discuss their audio technology. In the V30 they are using a ‘Quad DAC’ implementation, supporting 24-bit lossless audio output through the headphones. They are keen to promote that the V30 is the world’s first MQA mobile player for HiFi streaming – MQA is the Master Quality Authenticated audio codec using compression and file fingerprinting suitable for high fidelity digital audio from internet streaming. The HiFi modes on the phone are only available through the 3.5mm TRRS audio jack, and offer a small selection of EQ modes for different types of music. To assist with the audio experience, LG is partnering with Bang & Olufsen Play to provide some very reasonable in ear headphones.

I tried to do some non-blind testing while I was at the show, however all my lossless audio files are at home. What I did have to hand was entirely subjective – I couldn’t blind test, and the only real difference I heard was when I adjusted the EQ settings. LG doesn’t implement a full EQ configuration tool here, although I assume a third-party app could do the business.

For non-audio-visual features, LG is promoting its new voice recognition unlock. This allows the user to pick their own 3-5 syllable phrase to unlock the phone. Technically all Qualcomm Aqstic capable smartphones should be able to do this, and it allows unlock functionality in combination with face recognition, fingerprint recognition, and LG’s Knock code. The lock screen also provides a permanent display of the clock, the battery life, and any relevant notifications. The OLED panel we were told helps keep the power low with this feature, especially as the refresh rate can be super low (I imagine as low as one frame a minute, perhaps).

LG is also playing the AI game, by providing Google Assistant with the device. LG has worked with Google for LG specific commands, mainly for the camera, to allow certain voice commands to start recording video in wide angle mode, in Cine Log mode, or in manual mode. The idea being that the user can prop the smartphone up somewhere and tell it to start taking a video. Unrelated to Google Assistant, but LG will allow the camera to take an image when the user says the word ‘Kimchi’. I found this mildly amusing and very Korean.

LG talked a lot in our pre-briefing about color accuracy, mainly pointing at the camera technology, although they didn’t state what color accuracy they were aiming for on the display. The software does have an additional option to enable sRGB mode for photos, but in our preview device even at minor viewing angles we saw a very noticeable blue tinge.

For the design, LG is using a Gorilla Glass 5 back and front cover here; with a choice of Aurora Black, Moroccan Blue, Cloud Silver and Lavender Violet colors (some regions may not have all colors). The design is tested to MIL-STD-810G standards, known as MIL spec, and carries an IP68 rating, meaning that it is waterproof in 150cm / 5 feet of water for 30 minutes. That also typically means that manual repairs are a pain without warming up all the waterproof sealant in the device, unless LG is using O-rings to do the business. I’ll wait for iFixit’s teardown.

The LG V30 comes with a single USB Type-C port (no word if it is USB 2.0 or USB 3.0), and the power button is now part of the rear fingerprint sensor. Moving the power button from the side confused me for a strong minute, as the only other buttons on the device were the volume buttons on the left. The fingerprint sensor being on the rear is more suitable than one off to the side, and personally I’m not used to fingerprint buttons on the front. But in anecdotal testing the sensor recognition is a shade slower than my Mate 9, but not noticeably so unless you are concentrating hard.

LG is committed to bringing Android 8 over to the V30, and have us a tentative ‘probably within the next 6 months’ as an answer about moving on up.

This hands-on was a really small timeframe to test the handset – it was put in my hand at 10pm, I went to eat dinner, and it’s now 2am while I’m still writing it up. The press event for LG tomorrow starts at 8am for doors open, 9am for the presentation, so there’s not much to say yet aside from the general feel of the device (plus, as it stands I’ll get 4 hours sleep tonight). Personally, it’s going to be interesting – I’ve never used a phone with OLED or wireless charging yet, and with my current daily driver starting to feel the effects of slowdown, I’ll take this V30 for a spin for a few months and report back.

Gallery: LG V30 Hands On