server, cloned from https://git.disroot.org/mittorn/vulkan-playground
| android | ||
| companion | ||
| imgui | ||
| openxr | ||
| scripts/waflib | ||
| xpack | ||
| .gitignore | ||
| AGENTS.md | ||
| analyze_rate_control.comp | ||
| analyze_rate_control_finalize.comp | ||
| audio_opensles.h | ||
| audio_pipe.cpp | ||
| bitstream.h | ||
| block_packing.comp | ||
| comp.comp | ||
| compress-bc1.comp | ||
| compress_bc1.h | ||
| compress_xpack.h | ||
| config_test.cpp | ||
| constants.h | ||
| conversion.glsl | ||
| decoder_ffmpeg.h | ||
| decoder_mediacodec.h | ||
| decompose.comp | ||
| decomposer.h | ||
| dwt.comp | ||
| dwt_common.h | ||
| dwt_quant_scale.h | ||
| dwt_swizzle.h | ||
| filter.sh | ||
| fovcenter.vert | ||
| fovrec.frag | ||
| fovsimple.frag | ||
| fractal-image.comp | ||
| fxaa.glsl | ||
| gen_shader_bundle.py | ||
| glsl_shared.h | ||
| idwt.comp | ||
| idwt.frag | ||
| idwt.vert | ||
| image-decomposite.comp | ||
| image.comp | ||
| imgui_virtual_keyboard.h | ||
| LICENSE | ||
| nodecompose.comp | ||
| octogon.vert | ||
| octogon2.vert | ||
| octostrip.vert | ||
| patch_spirv_composite.py | ||
| positional_utl.h | ||
| pwtest.cpp | ||
| pyrowave_common.h | ||
| pyrowave_decoder.h | ||
| pyrowave_encoder.h | ||
| quad.comp | ||
| quad.frag | ||
| quad.vert | ||
| README.md | ||
| reconstruction.comp | ||
| reconstruction.frag | ||
| reconstructor.h | ||
| resolve_rate_control.comp | ||
| sgsr.frag | ||
| start_audio.sh | ||
| streamer_shared.h | ||
| triangle.frag | ||
| triangle.vert | ||
| tristrip.vert | ||
| vaapi-recorder-hevc.cpp | ||
| vaapi-recorder.cpp | ||
| vaapi-recorder.h | ||
| vaapi_encoder.h | ||
| vaapi_encoder_h264.h | ||
| vaapi_encoder_hevc.h | ||
| vkcapture.cpp | ||
| vkcompute.cpp | ||
| vkcompute2.cpp | ||
| vkgraphics.cpp | ||
| vkplayer.cpp | ||
| vkplayer2.cpp | ||
| vulkan_contructors.h | ||
| vulkan_framebuffer_utl.h | ||
| vulkan_pipeline_utl.h | ||
| vulkan_resource_utl.h | ||
| vulkan_texture_utl.h | ||
| vulkan_utl.h | ||
| waf | ||
| wavelet_dequant.comp | ||
| wavelet_quant.comp | ||
| wscript | ||
| xrapp.cpp | ||
| xrm_battery.h | ||
| xrm_config.h | ||
| xrm_counter.h | ||
| xrm_imgui_vk.h | ||
| xrm_ini.h | ||
| xrm_loader.h | ||
| xrm_log.h | ||
| xrm_openxr.h | ||
| xrm_protocol.h | ||
| xrm_shaders.h | ||
| xrm_solarxr.h | ||
| xrm_stats_shm.h | ||
| xrm_structreflect.h | ||
| xrm_utils.h | ||
| xrm_variant.h | ||
| xrmote_server.cpp | ||
| xrmote_server.h | ||
| xrmote_server_emulator.cpp | ||
| xrmote_server_gui.h | ||
vulkan-playground
xrmitter server grows here
Components
- server: Running directly on target device with OpenXR runtime. Receives layers and sound from clients and renders it. Contains all GUI settings
- streamer client: Driver or library, implementing openxr/openvr, receives tracking and streams video. There might be multiple client implementations (monado driver, monado in-process compositor, steamvr driver, OpenComposite driver)
- audio client: connects alsa/jack/pulseaudio to server with udp streams
- reconstructor/decomposer: stream encoding/decoding using Haar wavelet transform and video codec
Status
Server:
- TCP listener
- Receiving projection stream
- Connection/tracking with single client
- Very dumb audio streaming (does not use protocol)
- GUI prototype rendering (basic monitoring only, configuration does nothing)
- Android (Pico 4) and monado linux support (build scripts are not published yet)
- GUI server emulator (works without openxr runtime)
Clients:
- Audio with fifo pipes (tested with jack-stdio, should work with pulseaudio pipe sink/sources too)
- Monado driver with comp_target (monado interaction not published yet, WIP)
- Custom monado pacer synchronized with app for smoother framerate
- Monado builder with dynamic controller type switch (register all controller types)
Decomposer/Reconstructor:
- H264/HEVC VAAPI encoder (no ffmpeg used), linux only
- Haar decomposer compute pipeline
- BC1 reconstruction texture compression
- FXAA filter
- Android decoder
- ffmpeg decoder (software)
- Compute reconstruction pipeline (slow on Adreno)
- Graphics pipeline
- Foveated rendering (hard-coded scale)
Roadmap
Common:
- Choose better naming/branding
- PyroWave support
- Make reconstruction optional (allow using video only)
- Center position offset
- Refactoring, better project structure (split utilities)
- Choose build system
- Code style (it's AWFUL now!!!)
- Lost frame notification
- Imporve logging (use fmt_util if possible, remove extra
\n) - Shader combos to workaround SPIRV limitations
- Template combos (reduce unneeded branching)
- Improve vulkan utility
- New thread utility
- Optimize data layout to prevent false sharing
Server:
- Rec/Dec runtime configuration
- Configuration storage format
- GUI moving/toggle
- Improve gui monitoring (implot maybe)
- Interaction profile switches
- Emulate controllers in emulator GUI
- Timing information
- TCP streaming
- TCP keep-alive and reconnection
- UDP/mDNS broadcast
- SGSR/SGSR2 upscaling for reconstruction texture
- Foveated view configuration
- ffmpeg hwdec
- Multiple layers support (quad/cylinder layers)
- Depth/ASW in reconstruction data (16 bit or rgba block format)
- Disabling reconstruction (video rendering only)
Clients:
- Full jack/alsa/pulse audio clients
- Do something with shitty mediacodec on Android 14 (custom native window???)
- Make builder optional (restore static controller configuration)
- HMDInfo shared cache, so runtime can start without server connection
- Try send something from server to make ipv6 address availiable (udp broadcast???)
- Allow dynamic fov/resolution change (hack/bypass compositor?)
- UDP listener for autoconnect
- Monado compositor bypass (allow sending layers as is)
- Monado in-process compositor (do we need IPC server when we already have server on HMD?)
- Improve decomposer postprocessing
- Passing depth to server (Seems to be unsupported with OpenComposite/monado now)
- SteamVR driver
- Vulkan Video encoder
- External encoders (nvenc/amf/ffmpeg/x265) in separate process (TODO: can we use GPL software with IPC such way???)
- OpenComposite driver (at least, we can send textures without extra swapchain copy and update overlays without xrEndFrame call, so this should be useful)
- Windows support (at least in vulkan video/pyrowave mode)