{"id":222,"date":"2023-05-17T07:22:01","date_gmt":"2023-05-17T07:22:01","guid":{"rendered":"https:\/\/www.lasselaursen.com\/projects\/icon-set-selection-via-human-computation-3\/"},"modified":"2023-06-02T08:01:19","modified_gmt":"2023-06-02T08:01:19","slug":"planmixplay","status":"publish","type":"page","link":"https:\/\/www.lasselaursen.com\/projects\/planmixplay\/","title":{"rendered":"PlanMixPlay"},"content":{"rendered":"\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-1 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:66.66%\">\n<p>The elevator pitch for PlanMixPlay is a more engaging way of delivering an audio and video performance. More than 8 years in the making, PlanMixPlay is still in development and lining up to host a third prototype party in Berlin, Germany, the prior two taking place in Melbourne, Australia and Tokyo, Japan.<\/p>\n\n\n\n<p>A significant source of inspiration for PlanMixPlay is <a href=\"http:\/\/en.wikipedia.org\/wiki\/Soulwax\">2manydjs<\/a> and their <a href=\"http:\/\/vimeo.com\/25861552\" data-rel=\"lightbox-video-0\" data-magnific_type=\"video\">under the covers<\/a> tour. To this day, it remains a watershed moment in live audio\/video performances to me. However, at the time of writing, these types of interconnected audio video performances remain the exception rather than the rule.<\/p>\n\n\n\n<p>PlanMixPlay aims to try and pull impressive visual performances off the exclusive festival sized DJ dining table and on to the weekend club circuit.<\/p>\n\n\n\n<p>Furthermore, it serves as an experimental platform to try out novel live interaction patterns and strongly connected audio-video output.<\/p>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:33.33%\">\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"PlanMixPlay - Teaser 2 - NeonGrid Visualizer\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/50JxBTVGmi0?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"PlanMixPlay - Teaser 1\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/IyN5LuvvQlw?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n<\/div>\n<\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\">Publication and resources<\/h4>\n\n\n\n<ul>\n<li><strong><strong>A Multi-Touch DJ Interface with Remote Audience Feedback<\/strong><br><\/strong>Lasse Farnung Laursen, Masataka Goto, Takeo Igarashi, in ACM MM &#8217;14: The 22nd ACM International Conference on Multimedia, pp. 1225-1228<br><a rel=\"noreferrer noopener\" href=\"https:\/\/dl.acm.org\/doi\/10.1145\/2647868.2655065?cid=81554950756\" target=\"_blank\">Published PDF<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.lasselaursen.com\/projects\/social-dj\/\" data-type=\"page\" data-id=\"268\">Progenitor project &#8211; Social DJ<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/www.pechakucha.org\/presentations\/modern-touch-djing\">Pecha Kucha 20&#215;20 Tokyo Presentation: Modern Touch Djing<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/www.pechakucha.org\/presentations\/vj-dj-communication\">Pecha Kucha 20&#215;20 Tokyo Presentation: VJ DJ Communication<\/a><\/li>\n\n\n\n<li><a href=\"http:\/\/www.planmixplay.com\">Official PlanMixPlay Website<\/a><\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\">Quick Facts<\/h4>\n\n\n\n<ul>\n<li>8 Year on-going project<\/li>\n\n\n\n<li>Live prototype parties\n<ul>\n<li>14-06-2014 &#8211; Tokyo, Japan<\/li>\n\n\n\n<li>20-08-2022 &#8211; Melbourne, Australia<\/li>\n\n\n\n<li>TBD &#8211; Berlin, Germany<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Largest single-handedly designed application consisting of:\n<ul>\n<li>GUI library (soon to be replaced)<\/li>\n\n\n\n<li>Gesture detection engine<\/li>\n\n\n\n<li>Custom real-time audio engine (leveraging BASS)<\/li>\n\n\n\n<li>Custom real-time visuals rendering engine<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Built with modern C++11 &#8211; C++17<\/li>\n\n\n\n<li>Used in over 85 live performances<\/li>\n\n\n\n<li><a href=\"http:\/\/www.lasselaursen.com\/project\/planmixplay#variable-immediacy\">Variable immediacy<\/a><\/li>\n\n\n\n<li>Has its own <a href=\"http:\/\/www.planmixplay.com\/\">website<\/a>.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\">Connecting audio and video<\/h4>\n\n\n\n<p>Early on-site studies and interviews revealed that DJs and VJs rarely &#8211; if ever &#8211; communicate prior to a performance, in a weekend club setting. In the rare circumstance where they do communicate, it&#8217;s typically to ensure a thematic overlap, as opposed to on-going synchronization of audio and video. Furthermore, in numerous clubs DJs and VJs are placed far apart even prohibiting easy communication during a performance.<\/p>\n\n\n\n<p><a href=\"https:\/\/psycnet.apa.org\/record\/1996-06256-003\">Several<\/a> <a href=\"https:\/\/ieeexplore.ieee.org\/abstract\/document\/481694\">studies<\/a> have <a href=\"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2015.00736\/full\">investigated<\/a> the effects of non-synchronized audio and video. Humans vary in perceiving how out of sync visual and audio elements are, depending on the audio\/video content as well as whether the audio is too early, or too late, in relation to the video. In general, perceptions of overall quality improve the more synchronized audio and video are, effectively causing the most stimuli. In other words, well synchronized audio and video is always preferable and will stimulate audiences the most.<\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-2 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:66.66%\">\n<p>A high benchmark for a DJ performance with audio synchronized visuals is the &#8216;Under the Covers&#8217; live tour performed by 2manydjs. Having contracted visual artists to animate the cover art for a wide music selection, 2manydjs used a set of Pioneer DVJ-1000&#8217;s to perform fully synchronized audio video shows. The result is a striking and memorable performance that have few peers.<\/p>\n\n\n\n<p>The DVJ-1000&#8217;s enabled 2manydjs to playback audio with attached video files, with the same ease as standard CDJ&#8217;s allow a DJ to play and mix music. In other words, 2manydjs could effectively perform as &#8216;standard&#8217; DJs and the DVJ-1000&#8217;s would automatically produce visual output as a &#8216;side-effect&#8217; of their performance. This visual output was incredibly well synchronized and unique, particularly for first-time audience members.<\/p>\n\n\n\n<p>Their approach do have some significant drawbacks however:<\/p>\n\n\n\n<ul>\n<li>The visuals are hand-crafted to specifically suit one song (or occasionally 2 pre-mixed songs) and required considerable effort to create. Most people do not have the artistic skills or resources to pay artists to create custom made visuals for a large selection of songs.<\/li>\n\n\n\n<li>Given how each visual is custom made for one (or sometimes two) songs, they cannot be effectively re-used for any other songs.<\/li>\n\n\n\n<li>Not only are the Pioneer DVJ-1000&#8217;s &#8211; used by 2manydjs &#8211; expensive, they have also been discontinued.<\/li>\n<\/ul>\n\n\n\n<p>The success of their &#8216;Under the covers&#8217; tour highlight the value in synchronized audio video performances. PlanMixPlay&#8217;s <strong>core focus <\/strong>is improving live audio video performances.<\/p>\n\n\n\n<p>PlanMixPlay features &#8211; implemented and planned &#8211; to support this include:<\/p>\n\n\n\n<ul>\n<li>Implemented &#8211; Word-granular timing and visualization (<a href=\"https:\/\/www.lasselaursen.com\/projects\/lyric-manager\/\" data-type=\"page\" data-id=\"241\">and support tool<\/a>).<\/li>\n\n\n\n<li>Implemented &#8211; Multiple generative visualizers reacting to a range of audio frequencies.<\/li>\n\n\n\n<li>Partial Implementation &#8211; Artists compositional choices visualized as part of visual output.<\/li>\n\n\n\n<li>Planned &#8211; Audio stem separation to better visualize distinct portions of a song<\/li>\n<\/ul>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:33.33%\">\n<figure class=\"wp-block-embed is-type-video is-provider-vimeo wp-block-embed-vimeo wp-embed-aspect-1-1 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Under The Covers Vol. 1\" src=\"https:\/\/player.vimeo.com\/video\/25861552?dnt=1&amp;app_id=122963\" width=\"500\" height=\"500\" frameborder=\"0\" allow=\"autoplay; fullscreen; picture-in-picture; clipboard-write\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"PlanMixPlay - Auto-aligned Lyrics Visualization - Azealia Banks - Atm Jam Ft. Pharrell (Kaytranada)\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/_J1hhTWgCXM?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n<\/div>\n<\/div>\n\n\n\n<p>Two additional unique features of PlanMixPlay &#8211; Variable immediacy and remote audience feedback integration &#8211; are detailed below.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\">Variable Immediacy<\/h4>\n\n\n\n<p>Variable immediacy is the ability to separate action from reaction, i.e. extend the time between an event causing a reaction. In a performance context, this could be a DJ scratching a record and an audible record-scratch noise being emitted via attached speakers. Another example would be a drummer striking a drum and the sound being immediately audible from the drum itself.<\/p>\n\n\n\n<p>A close connected term is that of <em>liveness<\/em>, perhaps most deeply explored by <a href=\"http:\/\/homes.lmc.gatech.edu\/~auslander\/index.html\">Auslander<\/a>. <em>Liveness, <\/em>in a performance context, often refer to performances that have low (or fast) immediacy and utilize instruments that maximize expressiveness. Expressiveness is critical, as without it, one could use the definition to claim that a cd player is the epitome of liveness.<\/p>\n\n\n\n<p><a href=\"http:\/\/www.dtic.upf.edu\/~sergi\/\">Sergi Jord\u00e0<\/a> contributes a <a href=\"http:\/\/dl.acm.org\/citation.cfm?id=1085898\">concrete formula<\/a> defining the efficiency of a musical instrument thusly:<\/p>\n\n\n\n\n<p class=\"ql-center-displayed-equation\" style=\"line-height: 41px;\"><span class=\"ql-right-eqno\"> (1) <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.lasselaursen.com\/wp-content\/ql-cache\/quicklatex.com-37d0689e2940f59782305c06a40ab9ca_l3.png\" height=\"41\" width=\"598\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#98;&#101;&#103;&#105;&#110;&#123;&#101;&#113;&#117;&#97;&#116;&#105;&#111;&#110;&#42;&#125; &#77;&#117;&#115;&#105;&#99;&#73;&#110;&#115;&#116;&#114;&#69;&#102;&#102;&#105;&#99;&#95;&#123;&#99;&#111;&#114;&#114;&#101;&#99;&#116;&#125;&#32;&#61;&#32;&#92;&#102;&#114;&#97;&#99;&#123;&#77;&#117;&#115;&#105;&#99;&#79;&#117;&#116;&#112;&#117;&#116;&#67;&#111;&#109;&#112;&#108;&#101;&#120;&#105;&#116;&#121;&#32;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#80;&#101;&#114;&#102;&#111;&#114;&#109;&#101;&#114;&#70;&#114;&#101;&#101;&#100;&#111;&#109;&#125;&#123;&#67;&#111;&#110;&#116;&#114;&#111;&#108;&#73;&#110;&#112;&#117;&#116;&#67;&#111;&#109;&#112;&#108;&#101;&#120;&#105;&#116;&#121;&#125; &#92;&#101;&#110;&#100;&#123;&#101;&#113;&#117;&#97;&#116;&#105;&#111;&#110;&#42;&#125;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>\n\n\n\n<p>where<\/p>\n\n\n\n<p class=\"ql-center-displayed-equation\" style=\"line-height: 16px;\"><span class=\"ql-right-eqno\"> (2) <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.lasselaursen.com\/wp-content\/ql-cache\/quicklatex.com-91664d82ea72eb5cfdb455257e58c021_l3.png\" height=\"16\" width=\"528\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#98;&#101;&#103;&#105;&#110;&#123;&#101;&#113;&#117;&#97;&#116;&#105;&#111;&#110;&#42;&#125; &#80;&#101;&#114;&#102;&#111;&#114;&#109;&#101;&#114;&#70;&#114;&#101;&#101;&#100;&#111;&#109;&#32;&#61;&#32;&#80;&#101;&#114;&#102;&#70;&#114;&#101;&#101;&#100;&#77;&#111;&#118;&#101;&#109;&#101;&#110;&#116;&#32;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#80;&#101;&#114;&#102;&#70;&#114;&#101;&#101;&#100;&#67;&#104;&#111;&#105;&#99;&#101; &#92;&#101;&#110;&#100;&#123;&#101;&#113;&#117;&#97;&#116;&#105;&#111;&#110;&#42;&#125;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>\n\n\n\n<p>In very brief terms, his formula states that a musical instrument&#8217;s performance is maximized when its output complexity is maximized in conjunction with the performers freedom, and the input complexity is minimized. In other words, an efficient musical instrument is one that allows for a lot of flexibility both in terms of output and performer freedom, yet makes it as easy as possible for the performer to supply this input.<\/p>\n\n\n\n<p>With all that out of the way, we can finally return to <strong>Variable Immediacy<\/strong>. It enables performers  to separate themselves from liveness to a point they are comfortable with. For example, imagine that you&#8217;re a novice guitar player. Strumming a single note is difficult at first, but eventually possible to reliably complete in a timely fashion. As the number of notes that must be hit increase, as well as increase in complexity (chords), the act of playing the guitar becomes far more difficult. Learning to do this expediently and accurately is part and parcel of learning to play a guitar. But what if you could separate yourself from time? What if you could (seamlessly) move into the future and play a few chords, and return to right now and continue your playing. That is the core of variable immediacy. Separating performers from time, allowing them to build and perform more complex compositions.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\">Remote audience integration<\/h4>\n\n\n\n<p>The majority of live performance instruments do not integrate audience feedback. Partially because the most prolific and popular live performance instruments cemented their dominance in a time when this wasn&#8217;t feasible. The fact that integrating audience feedback hasn&#8217;t seen much traction in modern times can be attributed to several reasons:<\/p>\n\n\n\n<ul>\n<li>Many live performance instruments offer no straight-forward approach to do it. How would a guitar, drum, or turntable show any kind of feedback?<\/li>\n\n\n\n<li>Not all artists perform remotely. In local performances, there&#8217;s no sufficient need for instruments to support audience feedback. The artists can easily observe and hear the live audience feedback.<\/li>\n\n\n\n<li>Questionable value proposition. Some artists would describe the relationship with the audience as a love\/hate relationship. While general feedback &#8211; mostly positive &#8211; is appreciated, granular or negative feedback less so. Some artists see little value in a tight feedback loop during a performance.<\/li>\n<\/ul>\n\n\n\n<p>While the first two reasons are not debatable, the third one is. I believe that the rising trend of remote broadcasting\/streaming will continue and the landscape for remote audience integration has barely shifted in the past decade. Admittedly, it&#8217;s not an easy problem to solve. The bigger the audience, the more difficult it is to digitally condense a shared sentiment, in real-time. But if we look at popular contemporary solutions, such as a scrolling chat window, clearly there are improvements to be made. A scrolling chat scales incredibly poorly and leaves the performer with a limited impression of the audiences sentiment.<\/p>\n\n\n\n<p>I think there&#8217;s incredible potential in <a href=\"http:\/\/www.lasselaursen.com\/project\/social-dj\">integrating audience feedback directly into a performance interface<\/a>. There are &#8211; of course &#8211; many pitfalls to avoid, as well as careful considerations to make in terms of how to balance and best encourage interaction between the performer and the audience. But it&#8217;s a source of potential that I think is worth exhaustively exploring.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\">Acknowledgements<\/h4>\n\n\n\n<p>Numerous individuals were involved in both testing and providing helpful feedback as part of this project. I&#8217;d like to thank <a href=\"http:\/\/soundcloud.com\/nytelyfe\">Phil Huey<\/a>, <a href=\"http:\/\/www-ui.is.s.u-tokyo.ac.jp\/~nakajima\/index_en.html\">Makoto Nakajima<\/a>, <a href=\"http:\/\/iflyer.tv\/en\/vj-davidroy\/\">David Roy<\/a>, <a href=\"http:\/\/soundcloud.com\/djmodality\">Ryan Modality<\/a>, and Jan Rod for both helping in early interviews, participation in later user studies, as well as in the first private in-house party use of PlanMixPlay. Thank you also to Mark Jackson for participating in the latter event.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The elevator pitch for PlanMixPlay is a more engaging way of delivering an audio and video performance. More than 8 years in the making, PlanMixPlay is still in development and lining up to host a third prototype party in Berlin, Germany, the prior two taking place in Melbourne, Australia and Tokyo, Japan. A significant source [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":553,"parent":142,"menu_order":3,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_FSMCFIC_featured_image_caption":"","_FSMCFIC_featured_image_nocaption":"","_FSMCFIC_featured_image_hide":"","footnotes":""},"_links":{"self":[{"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/pages\/222"}],"collection":[{"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/comments?post=222"}],"version-history":[{"count":26,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/pages\/222\/revisions"}],"predecessor-version":[{"id":462,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/pages\/222\/revisions\/462"}],"up":[{"embeddable":true,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/pages\/142"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/media\/553"}],"wp:attachment":[{"href":"https:\/\/www.lasselaursen.com\/wp-json\/wp\/v2\/media?parent=222"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}