Audio and Video Puzzles Reference — Soft8Soft
Description: How to load, play and manage audio and video using Verge3D Puzzles.

# Audio and Video Puzzles

These puzzles are used to load and play back audio and video files, as well as to synthesize MIDI sounds.

## Contents

- Puzzles Reference
  - load sound
  - load video
  - play
  - pause
  - stop
  - rewind
  - set playback time
  - get playback time
  - get duration
  - set playback rate
  - is playing
  - set volume
  - mute
  - unmute
  - positional speaker
  - synth sound
  - note number
  - drum number

#

## Puzzles Reference

#

### load sound

Loads an audio file using a specified URL. The audio is played via the WebAudio API or HTML `<audio>` tag depending on the chosen type: sound/midi or music respectively. This puzzle also adds the loaded audio to a memory cache so that any subsequent uses of this puzzle with the same URL do not load the same sound file again. Using .mp3 format is recommended as it is supported by most web browsers.

For usage example, check out the Farmers Journey demo (source files available in the Asset Store).

#### preloading audio

Audio files take some time to load, so they are not available immediately. That's why simply using this puzzle inside the play puzzle may not work and the audio may not start playing.

In order to be sure that audio in your application works properly it's recommended to load audio files beforehand, for example, in the initialization tab which is executed before a Verge3D application is initialized. You can also use the connector puzzle to organize a bunch of "load sound" puzzles in a bit more convenient way:

Preloading audio in the init tab.

After that using the preloaded audio files either in the Main tab or user-created tabs won't cause any issues, since the audio is already cached after the first use of the "load sound" puzzle, hence it doesn't need to be downloaded again:

Using the preloaded audio in the main tab.

#### autoplay restrictions

In general, audio playback initiated programmatically (as in the case of puzzles) will most likely be blocked by a browser. This happens due to autoplay policies that restrict how audio can be played automatically on a web page. The purpose of such restrictions is to stop web pages from making spontaneous noises or playing audio when it's not desired, thus improving overall user experience.

Although, such policies can be implemented differently in various browsers and also can have various conditions for allowing / not allowing autoplay, the general rule for playing audio is to wait for the user to interact with your web page.

In cases where such interaction is already implied by the application's logic, for example, playing a sound on click (be it either a 3D-object or an HTML button), those restrictions are not what you should worry about. But, let's say that you want your application to have some background music that starts playing when the app is loaded. Since that can not always be done in an automatic way, a good solution would be to make a "launch" button for starting the app when it's loaded and ready. Clicking it counts as a user interaction and allows to start audio playback.

#### "sound" vs "music" vs "midi" audio type

This puzzle allows you to choose which API to use for loading and playing audio files.

sound (default)
  Suits better for short audio files, e.g. small sound effects; it's recommended to use this type only in the following cases:
  - Short looped (see the play puzzle and its loop option) sound effects — "sound" makes looped audio seamless, while with "music" you have a noticeable delay.
  - Sound effects that should be played immediately, e.g. on user interaction, experience a delay on Safari — "sound" fixes this issue.
music
  Should be used in all other cases; suits better for long audio files, e.g. background music.
midi
  Special type of sound that used to play Standard MIDI files (with .mid extension). Compared to music stored in .mp3, MIDI files are really lightweight and have minimum impact on app loading time. The downside of using MIDI — somewhat reduced quality of soundtrack.

#

### load video

Creates a HTML `<video>` element and loads a video file using a specified URL. This puzzle also adds the created video element to a memory cache so that any subsequent uses of this puzzle with the same URL do not load the same sound file again. Using .mp4 format is recommended as it is supported by most web browsers.

The load video puzzle does not create any visible elements on your scene. To make it visible, assign video texture to your materials using the replace texture puzzle.

For usage example, check out the Video Texture demo (source files available in the Asset Store).

#

### play

Starts audio or video playback. The loop checkbox repeats playback if enabled.

Due to browser restrictions (preventing malicious websites from playing noisy sounds and videos) you should not execute this puzzle to play your assets automatically (upon scene loading). Instead, you should do this in some user-generated event. Use when clicked or on event of to handle such events.

There is one exception to this rule. Browsers allow playing muted videos automatically. See the mute puzzle.

#

### pause

Pauses audio or video playback.

#

### stop

Stops audio or video playback. Using this puzzle is equivalent to using the pause puzzle, followed by the rewind puzzle.

#

### rewind

Makes the audio or video asset to play from the start.

#

### set playback time

Set audio/video current playback time in seconds.

Setting playback time is not supported for sound (WebAudio) assets.

#

### get playback time

Get audio/video current playback time in seconds.

Getting playback time is not supported for sound (WebAudio) assets.

#

### get duration

Get audio/video asset duration in seconds.

#

### set playback rate

Set audio/video playback rate. Specify value `<1` to slow down playback, `1` for normal speed, and `>1` to increase playback speed.

Setting playback rate is not supported for midi assets.

#

### is playing

Checks if a sound or video is being played back.

#

### set volume

Sets the sound volume. Input volume is clamped to the 0.0-1.0 range. This puzzle also works for videos with embedded audio track.

#

### mute

Mute a sound or video.

Apart from controlling audio volume, you can use this puzzle to start video playback automatically. This is possible, because web browsers allow playing muted videos without user interaction.

#

### unmute

Unmute a sound or video.

#

### positional speaker

Make an audio positional by attaching it to some 3D object.

Both sound and music audio assets can be turned to positional speakers. To fine-tune the 3D audio effect you can use the rolloff from – the distance from which the sound intensity starts to decrease, and the factor – the rate of this decreasing.

#

### synth sound

Play sound using the built-in General MIDI-compatible software synthesizer (based on the TinySynth library). Various instruments and effects supported.

note on
  Start to play note using the given synth channel with sound intensity specified as velocity (0-127). Use the at time value to specify playback starting time in seconds.
note off
  Stop to play given note.
set instrument
  Set channel instrument from the given list.
set volume
  Set channel volume.
set pan
  Set channel pan level.
stop sounds
  Stop all sounds on the given channel.

To play drums, specify Drums as channel number and use the drum number to select instrument.

For example, you can start making your piano model playable as follows:

For usage example, check out the Sound Synth demo (source files available in the Asset Store).

#

### note number

Return note number. Use it to pass notes to the 1-9 channel of the synth sound puzzle.

Octave numbers are specified using the scientific pitch notation, for example C 4 means middle C tone.

#

### drum number

Return drum number. Use it to pass notes to the Drums channel of the synth sound puzzle.

## Having Troubles with Puzzles?

Seek help on the forums!