Darren Mothersele - Drupal Planet Feed Darren Mothersele's Drupal Planet Feed http://www.darrenmothersele.com I Don't Use Recruitment Agents <p>I started working with Drupal full time in 2007. I knew back then I was on to a winner, as none of the other open-source systems I evaluated at the time offered the same power and flexibility. It took a while for mainstream web development community to catch on, but over the years the Drupal community has seen massive growth, and now Drupal powers some of the biggest sites on the internet, well over 1 million websites.</p> <p>But, this success brings problems, and one recurring complaint I&#39;ve heard over the years has been about the difficulty in finding top Drupal talent. This has made Drupal a prime target for recruitment agencies deception and dirty tricks.</p> <!--break--> <p>Wunderroot are a well known company in the Drupal world, and are known to be a good employer. As UK MD, Steve Parks, says in his blog <a href="http://wunderroot.co.uk/blog/we-dont-use-recruitment-agents">We Don&#39;t Use Recruitment Agents</a></p> <blockquote> We would really love to be able to use recruitment agencies — imagine: a team of people with genuine experience in hiring great staff, with fantastic contacts books, and taking the role of a trusted friend to guide us through advertising, filtering, selecting and engaging the right people. It'd be fantastic. We'd pay good money for that. <strong>Unfortunately, that's not how most recruitment agencies work in reality.</strong> </blockquote> <p>I have experience with working with recruitment consultants from both sides. Before I started freelancing in Drupal full time I was running a digital music startup. As a successful startup we experienced fast growth, and didn&#39;t have the resources in-house to do thorough candidate searches. We used a couple of recruitment consultants and were consistently disappointed. Candidates were misrepresented, to the point where one didn&#39;t recognise his own CV in an interview.</p> <p>On the other side, as a candidate, I do not use agencies for work. One experience in particular put me off for many years.</p> <p>I interviewed for a position, but decided after the first interview that, although the opportunity was interesting, I knew I was not the right candidate. The company wanted to invite me back for a second interview, but I told the consultant that I was not interested, and explained my reasons. Unfortunately, the consultant would not take no for an answer, and I was subjected to a week of harassment (to the point of bullying) over my decision.</p> <p>In <a href="http://wunderroot.co.uk/blog/we-dont-use-recruitment-agents">We Don&#39;t Use Recruitment Agents</a>, Steve Parks tells of a &quot;bait and switch&quot; operation where developers had been approached by recruitment agencies saying that they had been engaged by Wunderroot to headhunt (the bait) in order to get someone interested, but then saying the position was filled and proposing other positions (the switch).</p> <p>I&#39;m not sure if it&#39;s the same dirty tactic in operation, but I have heard in the past of an employer receiving my CV from an agency claiming to represent me. The employer knew me directly, so checked, and they had an out-of-date CV that I had given to the agency for a different opportunity previously. This came up in conversation at a Drupal meetup and it was suggested that this is probably not a mistake as other developers had heard of it happening too.</p> <p>The extreme of recruiters&#39; tricks are documented in <a href="http://web.archive.org/web/20120601080215/http://www.kernelmag.com/scene/2133/consol-yourself-with-this-one/">this old post</a> from Kernel Mag in which Consol Partners are accused of &quot;telling outrageous lies to candidates and start-ups&quot;.</p> <p>In a post on <a href="http://www.ere.net/2013/12/16/the-top-25-recruiting-trends-problems-and-opportunities-for-2014-part-2-of-2/">recruiting trends</a> ERE suggest that, in an era when candidate sourcing is becoming easier as everyone is &quot;findable&quot; on the internet, recruiters should &quot;shift toward improving the various selling components of recruiting&quot;. I&#39;m not sure exactly what they mean by &#39;<em>selling components</em>&#39; but I would beg recruitment agencies not to do this, and instead focus on providing <em>value</em>.</p> <h3>Recruiters - Do This:</h3> <p>Here&#39;s a short TODO list for recruiters:</p> <ul> <li>Clean up your industry: Get rid of the deception and bullying.</li> <li>Provide genuine value (c.f. Steve Parks quote above).</li> </ul> <h3>Until then...</h3> <p>If you&#39;re a reputable company looking to source Drupal developers, or you are a Drupal developer working in London or the UK, get in touch. I&#39;m starting a free job board on <a href="http://www.drupaldeveloper.co.uk/jobs">DrupalDeveloper.co.uk</a>.</p> Mon, 14 Apr 2014 00:00:00 +0100 http://www.darrenmothersele.com//blog/2014/04/14/no-recruiters http://www.darrenmothersele.com//blog/2014/04/14/no-recruiters Drupal Theme Generator Update <p>It&#39;s been a week now since I demoed my proof-of-concept for an automated theme generator at the Drupal show and tell event so I thought I&#39;d collect together the feedback I&#39;ve received so far and post an update.</p> <!--break--> <h3>Wrong Approach?</h3> <p>Almost unanimously positive feedback. In fact, it seems other people have been thinking along similar lines:</p> <blockquote class="twitter-tweet" lang="en-gb"><p><a href="https://twitter.com/mothersele">@mothersele</a> dude! just saw <a href="http://t.co/GyV2m41eUe">http://t.co/GyV2m41eUe</a> This is something that <a href="https://twitter.com/jenlampton">@jenlampton</a>, <a href="https://twitter.com/mortendk">@mortendk</a>, <a href="https://twitter.com/Cottser">@Cottser</a> and I have discussed for 8.x twig!</p>&mdash; Mark Carver (@mark_carver) <a href="https://twitter.com/mark_carver/statuses/450016685752201216">March 29, 2014</a></blockquote> <p>The one opposing view I have encountered wasn&#39;t actually against any of the ideas in the theme generator, but suggested that taking over Drupal markup was wrong and that we should be working with what Drupal provides. I know there are arguments for this, and if you want to go this route then you will need some other mechanism for documenting the conversion of your design to Drupal theme. If you want to argue this case, I&#39;d suggest first try having that discussion with <a href="https://twitter.com/mortendk">Morten</a>, as I&#39;m going to assume that we&#39;re all OK with the concept of taking complete control of (completely rewriting) Drupal&#39;s markup output.</p> <h3>Annotation</h3> <p>In an earlier prototype I had started working with annotations inside HTML comments but I found these increasing harder to parse as the extractions became more sophisticated. Someone in conversation brought up ideas from KSS and suggested looking at CSS comments as an alternative.</p> <p>I&#39;m still proposing this as a possible approach (see <a href="https://en.wikipedia.org/wiki/Docblock">Docblock</a>), but for now I&#39;m going to continue to annotate the markup (not the CSS) with x- attributes, as no one has had an issue with this, and at this stage it&#39;s easier to work with QueryPath to create the extractions based on these attributes. It seems that annotating the markup with x- attributes will be acceptable as long as they are stripped from the markup during the build process.</p> <blockquote class="twitter-tweet" lang="en-gb"><p><a href="https://twitter.com/rootwork">@rootwork</a> <a href="https://twitter.com/illepic">@illepic</a> <a href="https://twitter.com/micahgodbolt">@micahgodbolt</a> <a href="https://twitter.com/EvanLovely">@EvanLovely</a> <a href="https://twitter.com/mothersele">@mothersele</a> Interesting! Do the data attributes get stripped out during the build step?</p>&mdash; Brad Frost (@brad_frost) <a href="https://twitter.com/brad_frost/statuses/449662512624328704">March 28, 2014</a></blockquote> <p>It was great to get feedback from <a href="http://bradfrostweb.com/">Brad Frost</a> as his work on Atomic Design has been influential in the development of this process.</p> <h3>In code, or config</h3> <p>In this first proof-of-concept, the generated theme is held in memory, well actually it is persisted as a Drupal variable containing a single object that holds the result of all the &#39;extractions&#39; from the source. The original intention was that this would actually be a ctools exportable, so that it could be exported and managed as part of the configuration management process for the site.</p> <p>This is how the Panels flexible layout builder works. It has one parent layout plugin that programmatically declares child layout plugins based on the layouts you define using the layout builder tool. These child layouts are stored as exportable objects, so they can be exported using <a href="https://drupal.org/project/features">Features</a>. The current Hyde theme generator approach is similar, except that the parent plugins (for layout or styles) programmatically declare child layout and style plugins based on the result of each extraction from the HTML source design.</p> <p>Storing the result of the build in configuration or database raised some concerns, mainly over capturing the results in version control. These tweets summarise the issue:</p> <blockquote class="twitter-tweet" lang="en-gb"><p><a href="https://twitter.com/mothersele">@mothersele</a> interesting implementation. But I believe that should definitely generate theme in code, not just DB&#10;<a href="https://twitter.com/mcjim">@mcjim</a> <a href="https://twitter.com/MattFielding">@MattFielding</a></p>&mdash; Tom Bamford (@waako) <a href="https://twitter.com/waako/statuses/449539868411322368">March 28, 2014</a></blockquote> <blockquote class="twitter-tweet" lang="en-gb"><p><a href="https://twitter.com/waako">@waako</a> If a prototype is always in sync with a Drupal theme, the markup *is* all in code right? // <a href="https://twitter.com/mothersele">@mothersele</a> <a href="https://twitter.com/mcjim">@mcjim</a></p>&mdash; Matt Fielding (@MattFielding) <a href="https://twitter.com/MattFielding/statuses/449551562344386560">March 28, 2014</a></blockquote> <p>Matt picks up on my original intention, in that the design/theme would be captured in code and be version-able because the translation is automatic from the design&#39;s HTML/CSS/JS.</p> <p>The difficulty is in managing any changes that happen to the generated code once it becomes a Drupal theme. This is exactly the problem that using the theme generator is trying to solve. That it provides a documented, repeatable conversion process, so that design can become part of the (agile) development workflow.</p> <p>However, it is going to be unavoidable that some tweaking will be needed. This covers a couple more issues that were raised at the Drupal show-and-tell event:</p> <ul> <li>How to manage logic in template files?</li> <li>How to capture Drupal&#39;s pre-process functions?</li> </ul> <p>The approach I am looking at to solve this, is one I&#39;ve seen practised by other tools that involve code generation. For example, have you seen BDD using <a href="http://behat.org/">Behat</a>? When define a test scenario in Behat it generates stub code for any unrecognised steps in your tests. For example, if you say &quot;Given I am in a directory&quot;, you would get the generated stub code:</p> <div class="highlight"><pre><code class="text language-text" data-lang="text">/** * @Given /^I am in a directory &quot;([^&quot;]*)&quot;$/ */ public function iAmInADirectory($argument1) { throw new PendingException(); } </code></pre></div> <p>I think the theme generator could do something similar for elements marked as requiring pre-processing in the template file. This needs some further thought and perhaps a couple of experiements.</p> <h3>Terminology</h3> <p>Still struggling with naming conventions. If this is going to be a more general tool then need generally understandable terms (like &#39;component&#39;). But, need to avoid overloading terms even more, as it&#39;s already quite confusing having SMACSS modules, Drupal modules, panels, blocks, boxes, styles, layouts. urgh!</p> <h3>Next steps...</h3> <blockquote class="twitter-tweet" lang="en-gb"><p><a href="https://twitter.com/mothersele">@mothersele</a> <a href="https://twitter.com/mark_carver">@mark_carver</a> I love it. Also love that it works w/ panels! Q: Are the layout plugins placed in the theme? <a href="https://twitter.com/mortendk">@mortendk</a> <a href="https://twitter.com/Cottser">@Cottser</a></p>&mdash; Jen Lampton (@jenlampton) <a href="https://twitter.com/jenlampton/statuses/450703701263388672">March 31, 2014</a></blockquote> <p>So, I&#39;m going to revise the current proof-of-concept and produce a second prototype. This time as a Drush command that generates an actual Drupal theme. Rather than holding the extracted theme in configuration it will generate a theme folder, that will include all the usual Drupal theme files, plus any plugins for Panels layouts, styles, display suite etc, and the CSS/JS copied across from the source design.</p> <p>This will allow Hyde to generate stub code for pre-processing or other programmatic tweaks that are needed to get Drupal&#39;s output to match the design markup. I also think people will be more accepting of this approach as it&#39;s probably more like how it is expected to work.</p> <p>My worry is that people will then hack the generated theme, it will go out of sync with the source design markup, and that will break the whole process.</p> <p>If you want to get involved, please drop me a line. I need input from designers, themers, and developers. In particular, I&#39;d be interested to speak to anyone else already using Atomic Design and/or SMACSS on Drupal projects.</p> Fri, 04 Apr 2014 00:00:00 +0100 http://www.darrenmothersele.com//blog/2014/04/04/drupal-theme-generator-update http://www.darrenmothersele.com//blog/2014/04/04/drupal-theme-generator-update Drupal Theme Generator Demo <p>I&#39;ve been playing with the idea of automatically generating Drupal themes from static HTML/CSS/JS using annotations in the HTML markup. I put together a basic proof-of-concept of a tool to generate a Drupal theme, ctools layout and style plugins, and view modes and templates.</p> <p>Last night at the <a href="http://www.drupalshowandtell.com/">Drupal Show and Tell</a> event in London I gave a live demo of the theme generator in action. The event was recorded, so will be online eventually, but for now I&#39;ve recorded this demo as a couple of attendees suggested this would give a better idea of the detail that couldn&#39;t be seen on the screen during the live demo.</p> <!--break--> <p>My interest in this area came about through wanting to bring design into the development workflow of an agile project, and move away from the &#39;throw it over the fence&#39; mentality in design deliverables. You can read more about how this came about in my previous blog post <a href="http://darrenmothersele.com/blog/2014/03/12/automatic-drupal-theme-generation/">Death of the Themer</a>.</p> <h3>Assembly, not Deconstruction</h3> <p>Traditionally implementation of a design was done via a process of deconstruction from a PSD into flat HTML and CSS, and then another process of deconstruction in CMS implementation of the design. You can&#39;t design a responsive site in Photoshop so luckily this is changing. PSDs were horrible to work with as amends take far too long, and while Photoshop may be good to quickly mock up style ideas, pages designed in Photoshop tend rely too much on intuition, implications about how things would work, and tend to come with an implied &quot;you get the idea&quot;.</p> <h3>Atomic Design</h3> <p>As I&#39;ve mentioned in earlier posts, I&#39;m excited about the emerging trend towards atomic design (see <a href="http://bradfrostweb.com/blog/post/atomic-web-design/">Brad Frost</a>) as it brings a more &#39;development&#39; style process into design. Treating the process as that of designing a system of re-usable components, rather than just designing pages.</p> <p>This moves implementation of a design from a process of deconstruction, to a process of assembly, so brings the world of dev and design closer together. Either bringing design into the development workflow, or bringing development processes into design (depending on which way around you look at it).</p> <p>With an atomic approach to design, and with something like SMACSS for modular CSS, the process of converting to a Drupal theme can be automated. Because the markup/styles are &#39;componentised&#39; we can annotate the source code to document the conversion process and then use automated tools to manage the process.</p> <h3>Demo</h3> <p>Here&#39;s a demo of the proof-of-concept:</p> <div class='embed-container'><iframe src='https://player.vimeo.com/video/90315757' frameborder='0' webkitAllowFullScreen mozallowfullscreen allowFullScreen>https://player.vimeo.com/video/90315757</iframe></div> <h3>Next...</h3> <p>You can download/fork the code from the <a href="https://github.com/darrenmothersele/hyde">GitHub Hyde repo</a>. You&#39;ll need to patch the QueryPath module as it needs the latest version of the QueryPath library and the QP module doesn&#39;t include the right files to make this work by default.</p> <p>A lot of work needs to be done. This is very rough proof-of-concept code, but I think this shows the concept can work, and worthy of further development.</p> <p>Some feedback from last night included:</p> <ul> <li>Generate an actual theme. At the moment the theme is just an object stored in the DB/cache, but I had planned for this to be a ctools exportable. An earlier version I started working on generated actual theme files, perhaps it would be better to switch back to this approach?</li> <li>How to handle logic in template files? Shouldn&#39;t this be handled in pre-process?</li> <li>Stub code generation for pre-process functions</li> <li>Adding extra custom fields for display only? The example given was a date field that was displayed twice on page, once for date stored in field and once for time stored in same field.</li> </ul> <p>Drop me a line if you&#39;ve got any other ideas, or want to get involved, or want to help fund building this properly! :)</p> <h3>Update:</h3> <p>As requested, here&#39;s links to some of the other resources I mentioned during the Show and Tell...</p> <ul> <li><a href="http://webdesign.tutsplus.com/articles/the-sparkbox-responsive-design-process--webdesign-9651">Sparkbox Responsive Design Process</a></li> <li>Stephen Hay, quote: <a href="https://vimeo.com/47171001">We&#39;re not designing pages, we&#39;re designing systems of components</a></li> <li><a href="http://daverupert.com/2013/04/responsive-deliverables/">Tiny Bootstraps for Every Client</a></li> <li><a href="http://smacss.com/">Scalable and Modular Architecture for CSS (SMACSS)</a></li> <li><a href="http://patternlab.io/">Pattern Lab</a> and <a href="http://bradfrostweb.com/blog/post/atomic-web-design/">Brad Frost</a></li> <li><a href="http://warpspire.com/kss/">Knyle Style Sheets (KSS)</a></li> <li><a href="http://bem.info/method/">Block, Element, Modifier (BEM)</a></li> </ul> Fri, 28 Mar 2014 00:00:00 +0000 http://www.darrenmothersele.com//blog/2014/03/28/drupal-theme-generator-demo http://www.darrenmothersele.com//blog/2014/03/28/drupal-theme-generator-demo Dennis Dropin - BDD and Drupal <p>Last night I attended the <a href="http://www.meetup.com/DennisDropIn-Drupal-OpenSource/events/169988982/">Dennis Dropin</a> event on Behavior-Driven Development (BDD) and Drupal. This is a new event on the Drupal London scene hosted by Dennis Publishing.</p> <!--break--> <p>I&#39;ve been attending Drupal events in London since 2007, and there have been a wide variety of different types of events. There have been high points, such as <a href="http://darrenmothersele.com/blog/2013/03/05/drupal-camp-london-review.html">Drupal Camp London 2013</a>, and low points (we had quite a lot of Microsoft sponsored events, but one in particular springs to mind where every presentation was also either paid for by Microsoft or about Microsoft technology - I&#39;m not complaining - I have no right to complain as I&#39;ve eaten enough of their free pizza over the years!). From Drupal Camps, Drupalcon, code sprints, workshops, trainings, and just regular pub meetups, London has a really exciting and active Drupal-scene.</p> <p>Do we need another event?</p> <p>Well, if this first event in the series is anything to go by I&#39;d say definitely, yes please! The event took place in the boardroom at Dennis Publishing, so it was limited in capacity, about 25 people in attendance and the room was nicely packed. <a href="https://twitter.com/PaulLomax">Paul Lomax</a>, the CTO, did say that they had plans for a bigger space so by July we could see this expand into a bigger event.</p> <p>I can think of at least three reasons why this event was a success:</p> <ul> <li>It really felt like this was Dennis Publishing wanting to give something back to the Drupal community. The two presentations were perfectly matched. The first to motivate reasons for adopting BDD, and the second to give practical examples. There was enough technical detail to be useful, without being overwhelming.</li> <li>Dennis&#39; approach to digital is exemplary, probably due to Paul&#39;s leadership. It comes across that the whole team have the right balance of formal process and pragmatism.</li> <li>It seemed like a lot of the Dennis team were in attendance and the experience level of attendees was high. This meant there were some interesting discussions around the presentations too.</li> </ul> <p>The event had a natural split in two parts, which I&#39;d never really considered before, but perhaps others planning events could take into account. Pairing a presentation of motivating business cases with a more technical presentation of practical examples and inner workings.</p> <p>It was interesting to hear in the discussion alongside the presentations real world cases of how BDD had caught bugs that would have cost them ad revenue or resulted in financial losses if they had made it into production.</p> <p>I asked about Continuous Deployment and if introducing testing and increased the frequency of deployment. Paul confirmed they had been able to do more regular deployments, but gave a good reason (one I hadn&#39;t thought of) for why Continuous Deployment is not viable in a Drupal environment. His argument was that when you do a deployment you have to do a cache clear, and when you do a cache clear you clear the form cache. This kicks out any content editors that might be working on the site so deployments have to be limited and controlled. This lead to some discussion of a growl-like notification system with Node.js to push notifications to content editors logged in to the site.</p> <p>Another really interesting idea was the integration of visual diff into the Behat tests. This allows them to do a pixel-by-pixel comparison between the staging environment and production to flag up any major changes. A great extra precaution to pick up any bugs that may cause blocks to disappear or display incorrectly.</p> <p>The next event in the series, <a href="http://www.meetup.com/DennisDropIn-Drupal-OpenSource/events/169953872/">Responsive Web design: How Dennis are skinning this cat</a> is scheduled for April 16th. Already 17 people signed up to the Meetup group so I hope I get in!</p> Thu, 13 Mar 2014 00:00:00 +0000 http://www.darrenmothersele.com//blog/2014/03/13/dennis-dropin-drupal-bdd http://www.darrenmothersele.com//blog/2014/03/13/dennis-dropin-drupal-bdd Death of the Drupal Themer <p>At Drupal Camp London 2013 <a href="https://twitter.com/mcjim">James Panton</a> presented a methodology for implementing designs in Drupal without resorting to custom theme development. At this years Drupal Camp I pulled together two BoF (Birds of a Feather) sessions to continue the conversation and discuss prototyping designs, atomic design, and challenges of incorporating design activities into development workflow.</p> <p>In recent posts I&#39;ve talked about <a href="http://www.darrenmothersele.com/blog/2014/02/06/from-prototype-to-drupal-theme/">prototyping</a> and a bit about <a href="http://darrenmothersele.com/blog/2014/02/07/from-prototype-to-drupal-theme-2/">using Jekyll for prototyping</a>. In many ways this discussion is a continuation of those ideas, but also much more general in it&#39;s scope, and more wide-reaching in it&#39;s potential.</p> <!--break--> <h2>Death of the Themer?</h2> <p>I&#39;ve become interested in this area after recently facing challenges bringing design activities into the development workflow of an agile project. But first, for some context, let me explain where this all began...</p> <p>Last year, Jim presented a methodology for implementing designs in Drupal without having to do any custom theme development.</p> <p>First, Drupal is stripped of all it&#39;s markup. Jim presented to a packed room at Drupal Camp London, and received a big whooping round of applause when he showed naked Drupal markup. Perhaps indicating the general level of frustration the developers/themers have from battling with Drupal markup. Also, there&#39;s a general preconception that Drupal suffers very badly from &quot;Divitis&quot;, &quot;Classitis&quot; and &quot;Span-mania&quot;. This demo showed that this need not be the case.</p> <p>Next, markup is added back in bit by bit to match the markup in the flat HTML/CSS design. This means that the eventual markup produced by Drupal should exactly (or almost!) match what the designer originally intended. The rendered HTML is therefore cured of it&#39;s divitis (contains much less bloat), works with the CSS styles from the design with little or no tweaking, and can make use semantic HTML5 tags.</p> <p>The tools used to reset Drupal default markup include: <a href="https://drupal.org/project/semanticviews">Semanic Views</a>, <a href="https://drupal.org/project/ds">Display Suite</a>, <a href="https://drupal.org/project/panels">Panels</a>, and a fork of <a href="https://drupal.org/project/semantic_panels">Semantic Panels</a> that I can&#39;t seem to find a reference for. <a href="https://drupal.org/project/fences">Fences</a> can also be useful, although functionality to override field wrappers is also part of Display Suite.</p> <p>One of the main benefits of this approach is that it allows designers and Front-end developers to work freely, using whatever tools they want, and without consideration for any of Drupal&#39;s (perceived) limitations. There have been lots of really cool developments in front-end, including CSS and HTML preprocessors, and the multitude of CSS and JS frameworks. This approach allows designers, front end developers, UXs, and IAs to take advantage of all of these tools when prototyping sites.</p> <p>During the BoF sessions at this year&#39;s Drupal Camp London a few limitations and disadvantages were discussed. This methodology moves the implementation of design from development (writing code) to configuration. This can become quite repetitive as each individual bit of output has to be configured with appropriate resets and the correct wrappers to build up the markup.</p> <h2>Custom Plugins?</h2> <p>In the BoF sessions I presented a methodology that I had been playing with, inspired by Death of a Themer, whereby designs are implemented in a similar way, without the need for a custom theme. Drupal markup is taken to a &quot;full reset&quot; state, and then just the required markup is introduced through various plugins.</p> <p>This differs from the original methodology as it does not use configuration to add wrappers back in to fields, view modes, views, panels, etc. Instead, the markup is taken from the prototype (flat HTML and CSS) and copied into &quot;plugins&quot;. Various plugins are used:</p> <ul> <li>Display Suite plugins to add markup to content/entities in various view modes.</li> <li>Views styles to output list markup</li> <li>Panel&#39;s style plugins to add &quot;box&quot; markup, and,</li> <li>Panel&#39;s layout plugins to add layout markup.</li> </ul> <p>The result is the same markup as the flat prototype, and the same result as with the original Death of a Themer approach, but with much less configuration required.</p> <p>The real big win for this approach, is that the components of the design are reusable across various content types and sections of the site without having to repeatedly configure each occurrence.</p> <p>Building out these plugins for every bit of a design might seem like over-kill and more work than is really necessary, but it is a much more scalable way of theming as you end up with design components that can be reused across the site.</p> <p>This version of &quot;Death of a Themer&quot; means the Themer is replaced with a developer, rather than a site-builder, as the emphasis moves away from configuration towards writing custom plugins in code. However, I think a lot of the grunt work can be automated, as I&#39;ll explain in a bit.</p> <h2>Atomic Design</h2> <p>I saw parallels between this approach (of reusable theme components) and the emerging trend of <a href="http://bradfrostweb.com/blog/post/atomic-web-design/">atomic design</a>. In fact, I think atomic design is more than just a current trend but a real leap forward in design methodology.</p> <blockquote> <p>We&#39;re not designing pages, we&#39;re designing systems of components. <br> -- <a href="http://bradfrostweb.com/blog/mobile/bdconf-stephen-hay-presents-responsive-design-workflow/">Stephen Hay</a></p> </blockquote> <p>While producing an atomic design the designer is effectively breaking down the design into components. This is exactly what a good themer does when they take a design and decide how to implement it as a Drupal theme.</p> <h2>From Design to Theme</h2> <p>When a designer hands over a design (let&#39;s assume static HTML and CSS) the developer (or themer, or site-builder) has to make several decisions about how this will be implemented.</p> <ul> <li>Assumptions are made and things get &quot;lost in translation&quot;</li> <li>Compromises are made due to limitations in implementation (i.e. limitations of the CMS, content model, etc)</li> <li>Markup is augmented to add in extra properties like RDFa, etc</li> </ul> <h2>Agile</h2> <p>There shouldn&#39;t really be a &quot;handover&quot; of design. The style guide / atomic design or static mockup that is the result of prototyping should evolve as the product evolves.</p> <p>If something comes up that needs design we go back to the static HTML/CSS to implement it. If using atomic design, this means adding any required &quot;atoms&quot;, &quot;molecules&quot;, etc.</p> <p>If changes have been made as a design is implemented these need to be flowed back upstream to the style guide so that further development of the design can happen, and then (some of) the conversion process is repeated to bring across new design elements into the theme/templates.</p> <p>This process can be hard to manage and bring duplication of effort into the process.</p> <p>If we can automate some of the steps involved in going from static HTML and CSS to Drupal theme then we can move towards a more agile, and robust design-dev workflow.</p> <h2>Proof of concept</h2> <p>Automatically generating a Drupal theme from either a style guide, atomic design, pattern library, static mockup, prototype (or whatever you want to call it!) might sound like a crazy idea, but I&#39;ve got a working proof of concept.</p> <p>I decided to start from just pure HTML/CSS. After experimenting with static site generators (HTML preprocessors), and CSS preprocessors like SASS, I decided that it was best to take pure HTML/CSS as the starting point. This works because every design eventually ends up as HTML/CSS. It&#39;s the language of the web, so it&#39;s unavoidable. It means designers can use any combination of tools they like, as long as they produce HTML/CSS as the end result.</p> <p>As an example, I&#39;m using Jekyll to compile the static mockup, and Compass to compile the CSS, but this could be any combination of tools, or just hand-crafted HTML and CSS.</p> <p>The next stage is to add annotations into the source of the HTML to identify and describe each of the design &quot;components&quot;.</p> <p>I don&#39;t have any terminology yet. Drupal has it&#39;s own jargon, as does SMACSS, Atomic Design, OOCSS, KSS, etc. I&#39;m not sure what to call things. I&#39;m tending towards a combination of Atomic Design and Drupal terminology, but this will change.</p> <h2>Annotations</h2> <p>There has been some discussion in the design community about <a href="https://www.youtube.com/watch?v=ROaXVB-bbek">Pattern Library Annotations</a> documentation, etc.</p> <p>I want to use annotations in designs so that they can be automatically parsed and converted.</p> <p>In this proof of concept, annotations in the HTML source code map directly to the appropriate Drupal templates and plugins.</p> <p>I&#39;m using the <code>x-</code> prefix for my custom attributes. This mechanism is intended for use in vendor prefixing for vendor specific attributes. It distinguishes them nicely from <code>data-</code> custom attributes, results in valid HTML5, and therefore I think it&#39;s an acceptable abuse of the feature.</p> <p>For example, annotate a certain part of the source code as being responsible for layout and it will be automatically compiled into a Panels Layout Plugin:</p> <div class="highlight"><pre><code class="text language-text" data-lang="text"> &lt;div x-drupal-type=&quot;layout&quot; x-drupal-name=&quot;front_page&quot; x-drupal-label=&quot;Front page&quot; x-drupal-no-wrap&gt; &lt;div class=&quot;container&quot;&gt; &lt;div class=&quot;row&quot;&gt; &lt;div class=&quot;span8&quot; x-drupal-region x-drupal-name=&quot;content&quot;&gt; ... &lt;/div&gt; &lt;div class=&quot;span4&quot; x-drupal-region x-drupal-name=&quot;sidebar&quot;&gt; ... &lt;/div&gt; &lt;/div&gt; &lt;/div&gt; &lt;/div&gt; </code></pre></div> <p>Another example, this is content shown in a particular preview style called &quot;Box&quot;. It maps to a Drupal view mode:</p> <div class="highlight"><pre><code class="text language-text" data-lang="text"> &lt;div class=&quot;box&quot; x-drupal-type=&quot;display&quot; x-drupal-name=&quot;box_teaser&quot; x-drupal-label=&quot;Box teaser&quot;&gt; &lt;img src=&quot;&quot; width=&quot;100&quot; height=&quot;100&quot; alt=&quot;&quot; x-drupal-region x-drupal-name=&quot;image&quot; x-drupal-no-wrap&gt; &lt;h6 x-drupal-region x-drupal-name=&quot;title&quot;&gt;Easy to use&lt;/h6&gt; &lt;p x-drupal-region x-drupal-name=&quot;summary&quot; x-drupal-no-wrap&gt; To get started, you select the desired sample and base the entire website on it. It’s that simple! &lt;/p&gt; &lt;/div&gt; </code></pre></div> <p>Many other attributes are identified and the system currently supports panels layouts, panels styles, views styles, display suite (view mode templates), menus (kinda), and image styles.</p> <p>During the compilation process the various components are identified and the appropriate plugins are created. Any required static files, such as CSS and JS are copied across. Any content in the various regions is removed from the markup, and the generated plugins therefore only contain the markup to wrap the various parts of Drupal output.</p> <p>You can see the (very!) rough proof-of-concept here: <a href="https://github.com/darrenmothersele/hyde">https://github.com/darrenmothersele/hyde</a></p> <p>You really shouldn&#39;t try using this yet, but if you insist, install the modules on a Drupal site with Display Suite, Panels, etc, then enable both modules, and configure to point at the folder with your HTML/CSS design. Then click the &quot;Sync&quot; button to parse your design files and create and register the appropriate plugins based on the design.</p> <p>Warning: You are probably better off waiting for the next version or some actual documentation!</p> <h2>Rather see it in action?</h2> <p>If you would rather see it in action, then come along to the next <a href="http://www.meetup.com/drupal-show-and-tell/events/168733432/">Drupal Show and Tell</a> where I&#39;ll be demonstrating the early version of the theme generator and asking for feedback!</p> Wed, 12 Mar 2014 00:00:00 +0000 http://www.darrenmothersele.com//blog/2014/03/12/automatic-drupal-theme-generation http://www.darrenmothersele.com//blog/2014/03/12/automatic-drupal-theme-generation My Review of Drupal Camp London 2014 <p>Last weekend I was back at City University for the second Drupal Camp London. This year even bigger than the last, with over 600 attendees!</p> <p>I reviewed my post from <a href="http://localhost:4000/blog/2013/03/05/drupal-camp-london-review.html">last year</a> and it seems that I attended a lot fewer sessions this time around. That could be because I hadn&#39;t been to a Drupal event for a while, so I spent more time catching up with old friends and talking about the future of Drupal. I also ran two Bird of a Feather sessions on Atomic Design, Design Patterns, Death of the Themer, automated theme building, and general issues around bringing design into the development workflow, in particular for agile projects.</p> <!--break--> <h2>Friday</h2> <h3>Training day</h3> <p>Before the main Camp starts on Saturday there is the Business and Training day. The main CxO event is happening in another building, and has a strong line up of speakers. While this is going on I&#39;m down in the basement of City University with <a href="www.bluespark.com/team/ronald-ashri">Ronald Ashri</a> giving an &quot;Introduction to Drupal&quot; training.</p> <p>We&#39;ve actually got a large group with mixed abilities so we&#39;ve prepared a flexible training that covers a lot of material. Ronald prepared a comprehensive overview for the morning session. We went off-piste a few times, including some interesting discussions around configuration management and best practises. In the afternoon I prepared several demonstrations around an example site with content I had automatically imported from DBpedia. This covered content type configuration, simple Views configurations, some more advanced Views, and we even touched on creating custom workflows with Rules module. We had lots of good feedback about the training so I think this day was a success! A highlight was seeing how well my demo of the <a href="https://drupal.org/project/rules_link">Rules Link</a> module was received. Since I discovered this module I&#39;ve used it on every project, so it was good to share this with others.</p> <h2>Saturday</h2> <p>On to the main Drupal Camp...</p> <h3>Breakfast and chatting</h3> <p>After getting directions from the very hospitable Alex Elkins from City University, I arrive in time to find breakfast being delivered. There are several familiar faces around, so I take the time to catch up with people I haven&#39;t seen for a while.</p> <p>After leaving the canteen and heading out towards the rooms where the sessions are taking place, I made a stop in the &quot;sponsors room&quot;. This room seemed a bit strange to me. Why would sponsors pay a premium to be hidden away in a room off to the side? Last year you couldn&#39;t move for sponsors, they were all over the place! But I guess this is a side effect of the growth of the camp, and needing more space. Anyway, I decide to pop my head in to see who&#39;s around.</p> <p>I had a great chat with the guys from <a href="http://www.pulsant.com/">Pulsant</a> about their various options. At first I was surprised that they hadn&#39;t heard of Docker, but then perhaps not everyone is as excited about this as me. I&#39;m using Docker extensively now in <a href="http://www.apiarycloud.com">Apiary</a> and I do believe it will revolutionise our industry. Anyway, I quickly realised that this wasn&#39;t Pulsant&#39;s thing anyway, they are focused on the actually boxes and wires that run this stuff, and they seem to really care about getting that right. I&#39;ve seen them supporting lots of UK Drupal community events, so it was great to finally speak to them.</p> <p>I then got talking to iKos about the work they are doing with Drupal Commerce and SagePay. This is something I&#39;m probably going to be making extensive use of on a couple of projects, so I&#39;ll plug their free ebook <a href="http://i-kos.com/sagepay">Integrating Drupal, Commerce and Sage Pay</a>. Well worth a read.</p> <h3>Entity operations</h3> <p>All this chatting means I miss most of the first session and just make it in time for the end of Joachim&#39;s <a href="http://2014.drupalcamplondon.co.uk/drupalcamp-london-2014/session/fast-entities">presentation of entity operations</a>. I bump into an old colleague on the way out who&#39;s excited about going away and building some custom entities so I guess Joachim must have given a persuasive demo!</p> <h3>Demo Framework</h3> <p>Another coffee break, more chatting and I almost miss the next session, but determined to actually see something I rush off to find Annika&#39;s session on the <a href="https://drupal.org/project/df">Demo Framework</a>. I hadn&#39;t seen the Demo Framework before, but essentially, it looks like a distribution of Drupal with EVERYTHING in it, pre-configured to show off some of Drupal&#39;s sexiest features.</p> <p>From a technical point of view I really liked the combination of features and migrate module to create demo scenarios. With Features packaging up the functionality, and Migrate module used to package demo content.</p> <p>There was an amusing question from someone after this presentation who asked if it was dangerous to show clients all this functionality up front. They seemed concerned that it would then be hard to justify high prices after the client had seen all that Drupal could do. I think perhaps this person should attend one of <a href="http://www.acquia.com/about-us/team/jeffrey-jam-mcguire">Jeffrey McGuire&#39;s</a> sessions on selling the benefits of Drupal and open-source. My own opinion is that that you should always be providing value, and that when you provide value to a customer, it should not be hard to justify your price when it&#39;s based on the value you provide.</p> <h3>Lunch</h3> <p>Doesn&#39;t feel like I&#39;ve been here 5 mins and already I&#39;m eating lunch? It&#39;s no break from the Drupal action though, as Daniel from <a href="https://www.kendra.org">Kendra Initiative</a> has got a few people together to show off the mockup I made for a collaborative rights management tool for musicians. This is the start of a Technology Strategy Board funded project to develop an application to manage collaboration, metadata and rights, so I may well be blogging about this again in the near future.</p> <h3>Rules Rule Commerce</h3> <p>This session was pitched as an intermediate session, so I took along Alice, who is a relative newcomer to Drupal, but has been picking it up really quickly. Unfortunately, <a href="https://twitter.com/svenbergryen">Sven&#39;s</a> session was more of an introduction so there wasn&#39;t anything new here.</p> <h3>Harmony Forum</h3> <p>After leaving the Rules session I bumped into some people who had just come from a session around the corner on <a href="http://2014.drupalcamplondon.co.uk/drupalcamp-london-2014/session/harmony-forum-introduction">Harmony Forum</a> that sounded excellent. Drupal is lacking a good Forum option, so I was interested to hear what this was about.</p> <p>The most exciting thing in the open-source forum world at the moment is <a href="http://www.discourse.org/">Discourse</a>. I had looked at integrating Discourse with Drupal a while ago but not got very far with it. There&#39;s an existing Drupal module, but it proxies all requests via Drupal (which seemed like a bad idea from a performance and maintenance point of view) and it works against an old version of Discourse. I strongly believe in using the right tool for the job, and that not everything has to happen in Drupal, but this &quot;integration&quot; seems far too tightly coupled.</p> <p>I&#39;m undecided if integrating with Discourse, or building a modern forum solution natively in Drupal would be the best course of action. I think the answer will depend on the scale of the site in question. On smaller sites it makes sense to have everything under one roof, but I think for larger sites a separation of concerns offers extra scalability options that may be worth the overhead of supporting two systems. It&#39;s good that we potentially have both options.</p> <h3>Next Generation DevOps</h3> <p><a href="https://twitter.com/shrikeh">Barney Hanlon</a> gave us a tour of state-of-the-art DevOps in what he called the &quot;Five Stars of DevOps&quot;. Namely: Monitoring, Security, Performance, Automation and Scalability.</p> <p>As I tweeted at the time, this was a real highlight of the Camp. I&#39;ve been doing a lot of DevOps stuff recently, it&#39;s a really exciting area to be working and it&#39;s advancing quickly, so it&#39;s always good to hear what other people are up to. Barney presented this stuff really clearly and obviously knows his stuff. The slides from his presentation are available <a href="http://www.slideshare.net/shrikeh/devops-the-next-generation-slideshare">here</a>.</p> <p>In large-scale Drupal deployments I always encourage scaling out over scaling up. With effective DevOps automation, the pain of maintaining multiple systems is reduced, so you can benefit from using the most appropriate technology for each task. Barney summarised this with the question:</p> <blockquote> Am I doing something in my application that is done better by the infrastructure or an external service? <br><cite>- Barney Hanlon</cite> </blockquote> <p>I&#39;ve been making extensive use of Puppet throughout my DevOps work. It&#39;s based on Ruby, and I found the Puppet configuration files easy to work with. Barney highlighted something that had been bothering me about the Puppet approach, in that you need to have the Puppet Agent on the machine before you can provision it via Puppet. Not quite a chicken and egg style paradox, but it does mean you need someone other mechanism to first bootstap the machine and get the Puppet Agent on to it. Barney, instead recommends the use of <a href="http://www.ansible.com/">Anisble</a>. This is a pure SSH-based approach so needs no extra software on the machine to get started. I plan to move <a href="http://www.apiarycloud.com">Apiary&#39;s</a> initial bootstraping process to Ansible and still make use of Puppet, but then probably soon move all the configuration into Ansible.</p> <p>Barney presented an interesting stack he called the &quot;SPDY sandwich&quot;. It had multiple instances of Nginx with different responsibilities. I&#39;ve done this before for various reasons including load balancing and SSL termination, but Barney is using two layers of SPDY/Page speed optimisations. Perhap&#39;s now it&#39;s time to go and look again at supporting SPDY! <a href="http://blog.phusion.nl/2013/08/21/use-nginx-spdy-without-compiling-nginx-and-without-a-recent-openssl/">This blog post</a> seems like a good place to start, any pointers welcome!</p> <p>Barney re-iterates his point about not doing things in the application that can be done better by infrastructure, and he demonstrates this point using an example where he configures <a href="http://openresty.org/">OpenResty</a> to manage CSRF tokens. OpenResty is a web application server built on Nginx making use of the Lua scripting language. This opens up a whole new world of possibilities!</p> <p>And of course no presentation on DevOps would be complete without some talk of Docker. I think I&#39;ve said this before, but Docker is going to revolutionise our industry. Barney explained some of the current limitations (runs as root!) but Docker is still not ready for production and in heavy development. I did like the example scenario he gave where PECL dependencies could be installed in a Docker container and shared.</p> <h3>BoF - Death of the Themer</h3> <p>I finished the day with some impromptu BoF action. I was keen to get <a href="https://twitter.com/mcjim">James Panton</a> and <a href="https://twitter.com/jamiemagique">Jamie</a> into a room to discuss some of the ideas I&#39;ve been having recently around challenges of bringing design activities into a development workflow (in particular Agile).</p> <p>Last year James had presented on the Death of the Themer, and Jamie has been doing a lot of work with atomic design and live style guides. Unfortunately, they were coming to the Camp on different days, so I had to run two separate BoFs.</p> <p>I&#39;m going to write up the BoF notes and post them here separately. If you&#39;re at all curious about atomic design, style-guides, or automated Theme generation drop me a line, or look out for an announcement on Twitter.</p> <h2>Sunday</h2> <h3>Measuring Success (Piwik)</h3> <p>My training partner from Friday, Ronald, starts Sunday with a discussion of analytics. A lot of what he says applies to any website and analytics software combination, but I&#39;m particularly interested in the fact he&#39;s using Piwik. I&#39;ve been hearing a lot about Piwik recently, and I&#39;ve been meaning to check it out. You may remember I mentioned it on a post in my &quot;Cloud Survivalist&quot; series, as Google Analytics is the only Google service I have not yet found a replacement for.</p> <p>I&#39;d already decided, after talking to Ronald on Friday, that I was going to start using Piwik, but it was good to see it in operation. Ronald gave some good examples, including tracking content items wherever they appear, not just looking at page views. He did this by showing how to add tracking codes to content items in any view mode, so they are tracked in lists and views as well as full page view.</p> <p>Ronald shared a couple of tricks they have implemented to use learnings from analytics to drive content placement. He showed how you could use activity to automatically optimise content placement by doing things like moving items in a nodequeue on CRON based on reading data from the analytics API.</p> <h3>MTV</h3> <p><a href="https://twitter.com/iamreevo">Paul Reeves</a> gave an in-depth look into the guts of the new Drupal 7 powered platform at MTV. I&#39;d been involved in the architecture and development of this, but it was good to hear Paul explain the wider impact of the work we did.</p> <h3>More BoF...</h3> <p>The BoF continues from yesterday. As I said before, I&#39;ll post my notes up as a separate post, probably some time next week.</p> <h2>Wrap up</h2> <p>I might have attended fewer sessions than the previous year, but I left Drupal Camp feeling like the weekend had been a success. The sessions were just a small part of what Drupal Camp is about. It was more about the conversations, meeting up with old friends, making new connections, and throughout all this seeing common threads of conversation, perhaps indicating the current Drupal zeitgeist, namely...</p> <ul> <li>Drupal 8 is not going to be ready this year.</li> <li>Drupal 7 is the best it&#39;s going to be. D7 is a solid robust platform for developing on, and deserves a life beyond D8, a Drupal 7 LTS perhaps?</li> <li>Rules Link is an awesome module</li> <li>DevOps!</li> <li>Atomic design is more than a current trend, and there&#39;s an opportunity for design to become a more integral part of the development workflow.</li> </ul> <p>(or maybe that&#39;s just me).</p> <p>Finally, a big thank you to all the organisers: Tim Deeson, Hedley Smith, George Hazletood, Leon Tong, Ben Wilding, Alex Burrows, Della Deme, Farez Rahman, John Kennedy, (did I miss anyone?).</p> <p>Oh, and of course <a href="http://www.city.ac.uk/">City University</a>!</p> Wed, 05 Mar 2014 00:00:00 +0000 http://www.darrenmothersele.com//blog/2014/03/05/drupal-camp-london-2014 http://www.darrenmothersele.com//blog/2014/03/05/drupal-camp-london-2014 Drupal, Redis <p>The easiest performance boost for any Drupal site is to install Redis. This is a key-value store that you can use as a cache to drastically cut down on the number of database calls Drupal is making. These instructions are for Ubuntu 12.04.</p> <!--break--> <h3>Prerequisites</h3> <p>You need to install compilation tools as we&#39;ll be compiling the most recent version of Redis from source. TCL is also a requirement.</p> <div class="highlight"><pre><code class="text language-text" data-lang="text">sudo apt-get install build-essential sudo apt-get install tcl8.5 </code></pre></div> <h3>Download and Compile Redis</h3> <p>First download the most recent stable Redis version. At the time of writing this is 2.8.6. Extract the source code and compile. Do this as root:</p> <div class="highlight"><pre><code class="text language-text" data-lang="text">wget http://download.redis.io/releases/redis-2.8.6.tar.gz cs redis-2.8.6 make </code></pre></div> <p>After compilation it&#39;s a good idea to run the tests. You&#39;ll fail at this point if you didn&#39;t install TCL.</p> <div class="highlight"><pre><code class="text language-text" data-lang="text">make test </code></pre></div> <p>Then install Redis on your system, again as root:</p> <div class="highlight"><pre><code class="text language-text" data-lang="text">make install </code></pre></div> <p>There is a script provided that will install the Redis service on Ubuntu.</p> <div class="highlight"><pre><code class="text language-text" data-lang="text">sudo ./utils/install_server.sh </code></pre></div> <h3>Drupal configuration</h3> <p>First download the Predis PHP library for Redis into your sites Libraries folder. Predis library can be downloaded <a href="https://github.com/nrk/predis">here</a> and should be installed in <code>sites/all/libraries/predis</code>.</p> <p>Download an install the <a href="https://drupal.org/project/redis">Redis Drupal module</a>.</p> <p>Add the connection details to <code>settings.php</code> and the configuration required to allow Drupal&#39;s cache system to use Redis, rather than the database. Append this to <code>settings.php</code>...</p> <div class="highlight"><pre><code class="text language-text" data-lang="text">$conf[&#39;redis_client_interface&#39;] = &#39;Predis&#39;; $conf[&#39;redis_client_host&#39;] = &#39;127.0.0.1&#39;; $conf[&#39;lock_inc&#39;] = &#39;sites/all/modules/contrib/redis/redis.lock.inc&#39;; $conf[&#39;cache_backends&#39;][] = &#39;sites/all/modules/contrib/redis/redis.autoload.inc&#39;; $conf[&#39;cache_default_class&#39;] = &#39;Redis_Cache&#39;; </code></pre></div> <p>Now enjoy super fast Drupal!</p> Tue, 25 Feb 2014 00:00:00 +0000 http://www.darrenmothersele.com//blog/2014/02/25/drupal-redis http://www.darrenmothersele.com//blog/2014/02/25/drupal-redis Security, and Drupal <p>Did you know I provide various infosec services, including security audits of code and infrastructure, penetration testing, etc, for Drupal websites? It&#39;s not a subject that has featured in my blogs before, but having recently done many site audits, including reviewing sites for security issues, it&#39;s becoming a hot topic.</p> <p>Security consultant Egor Homakov recently posted up details of <a href="http://homakov.blogspot.co.uk/2014/02/how-i-hacked-github-again.html">how he hacked GitHub</a> and this got me thinking. Although I&#39;ve touched on security issues in previous posts (for example, <a href="http://darrenmothersele.com/blog/2013/12/19/drupal-common-mistakes/">common Drupal mistakes</a>), It&#39;s about time I posted something more comprehensive on the issue.</p> <!--break--> <h2>Don&#39;t ignore minor bugs</h2> <p>The most interesting thing about Egor&#39;s compromise of GitHub security was that he used 5 issues, which independently would have been very minor, but through clever combination of the 5 minor issues he crafted a high-severity exploit that could theoretically grant him access to all private repositories on GitHub.</p> <p>Egor reported the vulnerabilities under <a href="https://bounty.github.com/">GitHub&#39;s bounty programme</a>, so they could be fixed before details were released. He was awarded a bounty of $4000 USD for his work.</p> <h3>The cost of (in)security</h3> <p>Many of the <a href="https://news.ycombinator.com/item?id=7197048">comments on Hacker News</a> argued that this reward was far too low considering the sensitive nature of the compromise. Many large companies host important code-bases in private repositories on GitHub.</p> <p>In the past I&#39;ve found security issues (both minor and severe) on Drupal sites, and worked with third-party penetration testers to advise on recommendations on how to implement their findings.</p> <p>It&#39;s hard to measure the true cost of security issues should they be found and exploited on your site, but it should be obvious that investment in security review and penetration testing is a Good Thing.</p> <h3>White-hats</h3> <p>Egor reported his findings to GitHub so they could be fixed before details were released publicly. GitHub recently adopted a bounty program, which had already been pioneered by other large internet companies, like <a href="http://www.google.com/about/appsecurity/reward-program/">Google</a> and <a href="https://www.facebook.com/whitehat">Facebook</a>. These programmes aim to engage with &quot;independent security researchers&quot; (also known as &quot;white hats&quot;) to offer rewards for finding and reporting bugs.</p> <p>This reminds me of back in Oct 2008. I was using a service called &quot;blip.fm&quot; that aimed to be a &quot;Twitter for DJs&quot; but found performance issues with the player in my browser. Using some simple debugging tools and, with no effort, I uncovered a security issue that revealed all their private user data. I never actually managed to solve the player bug, but I did report the information disclosure issue to the site, to which they responded:</p> <blockquote>"Funny, we took the time to make all that kind of stuff private in our upcoming api and here we were passing stuff around in the clear right now - thanks for the heads up, we'll get this fixed." <cite>- Ryan (Blip.fm)</cite></blockquote> <p>I&#39;m not a security researcher, &quot;white hat&quot;, or any other type of hat. I don&#39;t regularly do &quot;security research&quot;, but I do know what mistakes to look for when reviewing a Drupal site and it&#39;s supporting infrastructure.</p> <h2>Security is hard</h2> <p>Further comments about Egor&#39;s post on Hacker News go on to talk about why security is hard.</p> <blockquote>I'm wondering if someone of Github's caliber can be hacked so easily, what about the rest of the masses developing web apps. - <cite> enscr (Hacker News)</cite></blockquote> <blockquote>Infosec is hard, but it is just one example of a bigger truth: Defense is hard. This comes up time and time again in any defensive discipline. The one peculiarity of InfoSec; you do not have any offensive capabilities. - <cite>dfc (Hacker News)</cite></blockquote> <h3>Minor flaws combine to make serious exploits</h3> <p>The GitHub vulnerability was found in their implementation of the <a href="http://en.wikipedia.org/wiki/OAuth">OAuth 2</a> standard. OAuth 2 is an authentication and authorisation protocol designed to grant certain 3rd parties (eg mobile apps or other web sites) controlled access to resources in your account on a service.</p> <p>It&#39;s a complex protocol and services often make mistakes in their implementation of the standard.</p> <p>What Egor&#39;s work shows is that even small, simple mistakes can be disastrous for security as when combined they can lead to much bigger issues.</p> <h3>No one gets it right every time</h3> <p>Everyone makes mistakes. No programmer has ever written perfect code, especially when working under the time and budget pressures of commercial projects.</p> <blockquote>Given a reasonably competent development team, you can usually make a first pass and find quite a number of low-hanging fruit security issues. Everyone makes mistakes, especially when under pressure to get a product out. Once those are gone you can use fuzzing and/or static analysis type techniques to find another set, but after that you get to the point where the bugs start getting quite obscure and require a fairly deep knowledge of how the system works so you can start stringing multiple problems together to get to a real security issue. - <cite>georgemcbay (Hacker News)</cite></blockquote> <p>You need to implement code review, and ideally hire third-parties to conduct audits.</p> <h2>Drupal security</h2> <p>Drupal has it&#39;s own <a href="https://drupal.org/security-team">Security Team</a> and <a href="https://drupal.org/node/101494">processes for reporting security issues</a>. The Drupal product itself has a good track record for security.</p> <p>But this only really covers the core of Drupal, and contrib modules. Almost all security issues I have found on Drupal sites arise from miss-configuration, custom themes, and custom modules. This makes sense, if you think about it, because that code does not get the same level of public scrutiny as the open-source code released on Drupal.org.</p> <p>Here are some pointers, to help with Drupal security. But, I highly recommend getting someone else to look over your code before pushing it into production.</p> <h3>Secure coding</h3> <p>According to the <a href="http://mdsec.net/wahh/">Web Application Hacker&#39;s Handbook</a> only a small percentage of website vulnerabilities today come from SQL injection. This is easy to protect against in Drupal by making correct use of the database abstraction layer. Check all custom code for calls to <code>db_query</code> to ensure it correctly passes in parameters to queries using the provided mechanism for escaping dangerous characters.</p> <p>More common vulnerabilities today come from Cross-Site scripting (XSS) and Cross-site Request Forgeries (CSRF).</p> <h4>XSS</h4> <p>XSS is the result of failing to take proper care when dealing with user input. Drupal provides many mechanisms for dealing with user-input, but you have to remember that Drupal takes a &quot;filter on output&quot; approach and always stores the original user input.</p> <p>This means that you have to always filter anything you output. Most sites I&#39;ve reviewed have missed this somewhere in their custom code.</p> <p>Drupal provides several mechanisms for doing this, but if they are missing from your code then you are doing something wrong. The functions to look for are:</p> <ul> <li><a href="https://api.drupal.org/api/drupal/includes!common.inc/function/check_url/7"><code>check_url</code></a></li> <li><a href="https://api.drupal.org/api/drupal/includes!bootstrap.inc/function/check_plain/7"><code>check_plain</code></a></li> <li><a href="https://api.drupal.org/api/drupal/modules!filter!filter.module/function/check_markup/7"><code>check_markup</code></a></li> <li><a href="https://api.drupal.org/api/drupal/includes!common.inc/function/filter_xss/7"><code>filter_xss</code></a></li> </ul> <p>Another essential requirement in protection against XSS is to ensure that you have correctly configured input formats.</p> <p>The <a href="https://drupal.org/project/better_formats">Better Formats</a> module can be useful to restrict input formats allowed in various places.</p> <p>Any input filter, beyond the plain text filter, that you assign to a user should be double checked to ensure it&#39;s configured securely. That means ensuring that any HTML codes are filtered. You may want to allow a small set of safe tags for basic formatting, but never allow image tags to any user you wouldn&#39;t trust with full admin access to the server.</p> <p>There is no difference if image tags are entered by the user, or via a WYSIWYG editor, they are still a big security hole. If you absolutely must have users posting images ensure they are always correctly filtered and always hosted from your server. The <a href="https://drupal.org/project/image_resize_filter">image resize filter</a> can help with this.</p> <h4>CSRF</h4> <p>CSRF exploits a user&#39;s logged in session by forging requests to the site that are initiated without the user being aware. If you have a GET request that causes some state change in your app or data, then you have the potential for a CSRF exploit. GET requests are not supposed (by the HTTP standard) to allow for this anyway, but, if you do need a state changing GET request you have a couple of options:</p> <ol> <li><p>Use a tokenized URL, for example, the flag links generated by Drupal&#39;s Flag module. Ensure that the links are generated with a unique token so you can identify the resulting request and confirm that it originated from a link that you generated. Luckily the Drupal API (when properly used) can do this for you.</p></li> <li><p>Link to a confirm form, which uses a POST request to actually execute the change in the system. You see this all through Drupal core, for example when deleting a node. The Drupal API has functions to make simple confirmation pages.</p></li> </ol> <p>CSRF is also possible in POST requests, but you are protected by the automatic CSRF protection in the Drupal Form API. Make sure that all forms on your site are generated and processed correctly. There&#39;s an old docs page <a href="https://drupal.org/node/178896">here</a> as a starting point.</p> <h4>Access bypass</h4> <p>All custom code must use the proper mechanisms to ensure controlled access to resources. The Drupal API provides various mechanisms to do this, including:</p> <ul> <li>The Drupal menu system defines routes from URL paths to actual callbacks. It includes an access callback mechanism to check permissions before allowing access to the URL.</li> <li>The <code>user_access</code> API function can be used to check the current user has a specified permission.</li> <li>The node_access API should be used to correctly check access to content. This can be done in custom database queries by using <code>$query-&gt;addtag(&#39;node_access&#39;)</code></li> <li>Code that works defines custom entity types should also define their own access callbacks.</li> </ul> <p>A big mistake I have seen a couple of times in custom code is to accidentally assign to the global user object. In one particular case a faulty <code>if</code> statement in a template file was giving every user who accessed a specific node admin access. This is possible the most dangerous typo I have ever seen in over 20 years of programming!</p> <div class="highlight"><pre><code class="text language-text" data-lang="text">global $user; if ($user-&gt;uid = 1) { echo &#39;hello admin&#39;; } </code></pre></div> <p>That&#39;s not the actual code, but a similar example. Obviously the programmer had meant to write <code>if ($user-&gt;uid == 1)</code>, which is a check for equality. There were many other errors in the code which made this situation much worse. My recommendations were:</p> <ul> <li>Move logic out of template files into a preprocess function.</li> <li>Don&#39;t access the global <code>$user</code> object and instead pass an <code>$account</code> variable as a parameter to the template file.</li> <li>Don&#39;t mess with the global <code>$user</code> object.</li> <li>Oh, and don&#39;t mess with the global <code>$user</code> object.</li> </ul> <h3>Passwords</h3> <p>You may have several different levels of user account (different roles) all with different levels of authorisation. Enforcing strong passwords across all accounts is essential. A compromised user account with hardly any permissions combined with a privilege escalation vulnerability could combine to lead to full pwning of your site and server.</p> <p>There are modules available to help secure user accounts and sessions to prevent unauthorised access, implement a password policy and restrict failed login attempts:</p> <ul> <li><a href="https://drupal.org/project/single_login">single_login</a> will detect and prevent duplicate logins on the same account.</li> <li><a href="https://drupal.org/project/password_policy">password_policy</a> allows you to specify a set of constraints that all passwords must match.</li> <li><a href="https://drupal.org/project/flood_control">flood_control</a> is built into Drupal 7 but this module provides an admin UI for it.</li> <li><a href="https://drupal.org/project/login_security">login_security</a> add various options to prevent the brute forcing of user accounts.</li> </ul> <p><strong>Update (6th March &#39;14):</strong> <a href="https://twitter.com/mgifford">Mike Gifford</a> has suggested that perhaps the <a href="https://drupal.org/project/session_limit">Session Limit</a> module might be a better option instead of Single Login. I have to agree it does look like a better option, thanks Mike!</p> <p>Improving UX can also help in security. Password requirements, failed logins, failed registration attempts, etc all contribute to bad user experiences. There are a few Drupal modules to help with this:</p> <ul> <li><a href="https://drupal.org/project/password_tab">password_tab</a> moves the password forms out to a separate tab/page in the user account.</li> <li><a href="https://drupal.org/project/prlp">prlp</a> (or password reset landing page) fixes a common source of user support requests from lost passwords - this is how core password reset should really work!</li> </ul> <h3>SSL</h3> <p><strong>If you are not logging in to your site via HTTPS they you are handing out your password to anyone on the same network as you.</strong> If any of your users access the site via a public wifi this should be a concern. It&#39;s easy to pick up unencrypted network traffic with applications like Wireshark or a WiFi Pineapple.</p> <p><strong>Probably the biggest issue I am seeing today is the lack of SSL in the Drupal community.</strong> I don&#39;t understand why every site is not using either <a href="https://drupal.org/project/securelogin">securelogin</a>, <a href="https://drupal.org/project/securepages">securepages</a> or some other method to ensure that all logins and passwords are sent encrypted.</p> <p>Ensure you have the server correctly configured to deliver over HTTPS before enabling secure pages. Also be aware that serving over HTTPS is more resource intensive than HTTP.</p> <h3>Server issues</h3> <p>Securing your servers is a whole other story, and could take up several blogs worth of information. Here&#39;s the basics:</p> <ul> <li>Your production (live) site should run on a separate environment and the only ports open on the machine should be 80 and 443, for HTTP and HTTPS traffic.</li> <li>The rest of your environment should be strongly firewalled. This includes all the backing services such as Databases, Redis cache, Solr search etc, management applications like version control servers, and development and QA environments.</li> <li>Avoid insecure protocols like FTP.</li> <li>The server OS and all other software needs to be kept up-to-date.</li> </ul> <h3>Information disclosure</h3> <p>This is a common issue that is picked up by penetration testers employed to do testing on Drupal websites. There are many ways that a Drupal website discloses more information that it needs to. Including:</p> <ul> <li>HTTP headers revealing Drupal version, Apache/Nginx version, and PHP software versions.</li> <li>robots.txt revealing admin URL locations</li> <li>Metatags disclosing what software is being used</li> <li>Drupal&#39;s default login error message confirms the existence of accounts.</li> </ul> <p>Perhaps the most revealing information disclosure issue is with sites that leave the <code>CHANGELOG.txt</code> file visible to the public. This file reveals the exact version of Drupal that you are using, and from there it&#39;s easy to work out what vulnerabilities your site could be open to.</p> <p>My recommendation is to rewrite the location of <code>CHANGELOG.txt</code> and other unnecessary files to a 404 error in the server config. This is better than removing the files, as they could be accidentally restored during future upgrades. The Apache config rules to do this are quoted in <a href="https://github.com/darrenmothersele/aegir_secure_mods/blob/master/provision/securemods.drush.inc">this file</a>.</p> <h3>And finally...</h3> <p>Turn off access for user 1. The super admin user who can do anything is never required on a production site. Turn it off by setting status=0 in the database.</p> <p>That is just a whirlwind tour of Drupal security, but it should be enough to help you do a first-pass for obvious security issues.</p> Thu, 20 Feb 2014 00:00:00 +0000 http://www.darrenmothersele.com//blog/2014/02/20/drupal-security http://www.darrenmothersele.com//blog/2014/02/20/drupal-security From Prototype to Drupal (part 1) <p>The wrong way to build a Drupal site is to jump straight into Drupal and start building out content types. I&#39;ve heard a lot of people praise Drupal for it&#39;s usefulness in prototyping, and I&#39;ve been guilty of doing this myself in the past, but this is not a good idea for many reasons.</p> <p>In this first post of the series I explain why I prototype and then introduce some tools to help with prototyping. In future parts will look at the process of building a prototype in a way that supports effortless translation into a Drupal website.</p> <!--break--> <h2>Why prototype?</h2> <p>I think the biggest danger of jumping straight into Drupal without a prototype is that you get locked into Drupal ways of doing things too early in the process and you&#39;ll be naturally thinking in Drupal terms, and end up with a &#39;drupaly&#39; solution. You miss out on other cool ideas or possible solutions that might be a bit harder to implement, but result in a much better product.</p> <p>Also, if you start prototyping in Drupal, or building in Drupal straight away, then you end up bringing across elements from prototypes into your final site build. This big ball of mud grows and can become a maintenance nightmare. What you really want at the prototype stage is the freedom to throw things away. You shouldn&#39;t feel tied to any idea, design, or solution, which will happen if you&#39;ve invested time and effort in building it. Prototypes should be good to throw away at any point and start again.</p> <p>The best way to prototype web sites is to simply build out some HTML and CSS. You can quickly mock up ideas, test it out on users, and iterate towards a final design. Lots of other <a href="http://deeson-online.co.uk/blog/capturing-feedback-prototypes-drupal">developers</a> and <a href="http://frontendunited.org/session-info/prototyping-your-way-ux-success">UX experts</a> agree with me on this one.</p> <h2>Prototyping tools</h2> <p>There are short-cuts to building out flat HTML and CSS that, while not necessarily intended for prototyping, certainly speed up the process.</p> <h3>Front-end frameworks</h3> <p>The benefits of using a framework like <a href="http://getbootstrap.com/">Twitter Bootstrap</a>, <a href="http://foundation.zurb.com/">Zurb Foundation</a>, or Yahoo&#39;s <a href="http://purecss.io/">Pure CSS</a> is that they give you building blocks you can use to quickly piece together a responsive website. Each of them have their own way of doing things, so you must be careful not to limit your thinking so you don&#39;t end up implementing things a certain way just because that&#39;s how the front-end framework does things.</p> <p>Pure CSS offers a very minimal starting point so you have to do more of the work, but this will lead to more varied solutions. Twitter Bootstap is much more comprehensive, but without a lot of customisation you&#39;re likely to end up with something that looks like it was built using Twitter Bootstrap.</p> <p>I find that Zurb Foundation offers a good compromise, being a fully-featured front-end framework, but having lots of scope for customisation using SASS. There are settings variables to control lots of the default behaviour, and by using the SASS mixins you can easily use the features of Foundation with your own markup.</p> <p>In the early prototype stages I tend to make more use of the pre-defined classes and styles from Zurb, replacing them with custom classes using the mixins as the prototype evolves.</p> <h3>HTML Preprocessor</h3> <p>When creating a prototype site in HTML you will end up repeating yourself and writing the same HTML code multiple times. Using a HTML preprocessor helps with this as you can include special tags in your HTML code which the preprocessor replaces with other HTML. This cuts down on the time taken to write the HTML in the first place, and speeds up the process of making changes to your prototype. For example, if you have a navigation bar that is common to all pages of your prototype you define it once and then just include it in every page. Your prototype is then built out of various components, which makes it easier to manage and quicker to iterate.</p> <p>I&#39;ve talked about a HTML preprocessor <a href="http://middlemanapp.com/">Middleman</a> in a previous <a href="http://www.darrenmothersele.com/blog/2013/08/02/cms-is-dead-long-live-cms/">blog post</a> where I discussed using one as a static site builder. Back then I came out in favour of Middleman, but since then I got turned on to <a href="http://jekyllrb.com/">Jekyll</a> when I migrated my site to GitHub pages, and I&#39;ve been in love ever since.</p> <h3>CSS Preprocessor</h3> <p>Just like a HTML preprocessor speeds up the creation of HTML for your prototype, a CSS preprocessor speeds up the creation of CSS.</p> <p>I&#39;ve been using <a href="http://sass-lang.com/">SASS</a> and <a href="http://compass-style.org/">Compass</a> for a while, as it fits well with my workflow but there are other options, such as <a href="http://lesscss.org/">LESS</a>.</p> <p>There&#39;s an excellent starting theme for Drupal called <a href="https://drupal.org/project/aurora">Aurora</a> that provides a mechanism for incorporating SASS and Compass into a Drupal development workflow. But, we avoiding jumping straight into Drupal site building. From a prototyping point of view, the main reason I&#39;m using SASS/Compass is so that I can make use of Zurb Foundation mixins.</p> <h2>Next time...</h2> <p>In the <a href="http://www.darrenmothersele.com/blog/2014/02/07/from-prototype-to-drupal-theme-2/">next post of this series</a> I&#39;m going to show you how to create an all-in-one prototyping tool that will work for all stages of prototyping, from the throwaway early drafts, through later refinement stages, up to the final stages of prototype conversion to Drupal.</p> Thu, 06 Feb 2014 00:00:00 +0000 http://www.darrenmothersele.com//blog/2014/02/06/from-prototype-to-drupal-theme http://www.darrenmothersele.com//blog/2014/02/06/from-prototype-to-drupal-theme Some Common Drupal Mistakes <p>Over the past few months I&#39;ve had the privilege of being asked to audit several Drupal websites, both large, small, and very large. This is a collection of common mistakes I&#39;ve spotted. This is by no means an extensive check list, and you may disagree with some of my conclusions, but I&#39;m posting this here in the hope that it is of help to the community.</p> <!--break--> <p>Every developer loves code review right? I know I do, as it&#39;s always interesting to get someone else&#39;s point of view, and always leads to better code. It&#39;s impossible for a single developer to produce quality code. That&#39;s an immutable law of software engineering. In an ideal world we&#39;d all have an infinite supply of inspiration, attention span, time, and client patience. But, in the real world we make compromises, because either budgets are tight, time is short, or scope is just too wide ranging.</p> <p>These are some observations I&#39;ve made over the past few months while performing code review, site audits, or security checks.</p> <h3>Configuration Management</h3> <p>One of the main issues with Drupal (which is being addressed in Drupal 8) is the lack of separation between content and configuration. It is important to distinguish between the two in order to manage configuration separately to content. Ideally it should be possible to construct a bare version of the site without content using just the codebase as all configuration should be captured in code. There are several approaches to this in Drupal including the popular <a href="http://drupal.org/project/features">Features</a> module.</p> <h3> Unnecessary module use</h3> <p>Adding extra modules to a Drupal site adds overhead. The extra source code has to be parsed by PHP and modules add hooks which can expand the call graph for every page request. With each extra module you&#39;re adding another dependency to keep updated and maintained. Some modules add 100s of kilobytes of source code for a bit of functionality that could have been added with a few lines of code.</p> <p>Sometimes modules are left hanging around from earlier in development. You may have installed a module to evaluate it but decided not to use it. Make sure it&#39;s uninstalled (remove database tables) and deleted from the codebase.</p> <p>Also, remember to disable any developer modules on the production site. You don&#39;t often need the Views UI, devel and many other administration modules on a production site.</p> <h3> Views issues</h3> <p>Views causes loads of issues. Out of all the sites I&#39;ve looked at recently, none of them had adequate configuration of the Views module. It&#39;s all to easy for site builders to create lots of extra Views, but many neglect to enable caching. You should at least enable a short query cache time. You can set a long rendered output cache, because it gets invalidated every time the query reruns anyway.</p> <p>More often than not, anything but the most basic View does not have adequate indexing defined in the database. I think you&#39;d be surprised when you look into the MySQL slow query log and see how many times the database has been forced to filesort. This is a performance killer and it means that MySQL didn&#39;t have appropriate indexes for the query, and had to result to writing out to disk to sort the results.</p> <p>Consider replacing some views with blocks and a direct database query. This has it&#39;s advantages at times.</p> <h3>Diff analysis</h3> <p>If you&#39;ve got access to Drush you can generate a make file from a working Drupal site, then run Drush Make to recreate a clean copy of the codebase.</p> <p>Then use the diff tool to see differences between the two code bases. This will highlight if any of the Drupal core or contrib module files have been hacked.</p> <h3>Full HTML input filter</h3> <p>A very popular mistake is giving away permission to post HTML code to your site. This is really dangerous. Only the most trusted users should be given permission to post HTML, and even then it should be filtered to just the set of tags they actually need. Even an image (img) tag in the wrong hands is a security nightmare and will lead to XSS and other issues.</p> <h3>Output of unsanitised user input</h3> <p>Remember that Drupal&#39;s approach is to store raw user input and sanitise it on display. Never output any user generated text without first running it through one of Drupal&#39;s sanitisation functions. Double check any custom code and template files for <code>echo</code> or <code>print</code>ing of fields without sanitisation.</p> <h3>PHP Errors</h3> <p>I often see lots of PHP errors building up in the watchdog table. This is bad, not only because it indicates there is some coding error on the site, but it&#39;s bad even if the error isn&#39;t manifesting itself in any visible effect on the site, because it locks the watchdog table every time it logs an error. This is a performance killer, so clear up those PHP errors!</p> <p>You probably don&#39;t want to use database logging on a production site anyway. I&#39;d recommend logging to syslog and then moving logs out to an central place, like elasticsearch or similar.</p> <h3>Conclusion</h3> <p>These are just a few of the issues that I commonly see on sites, and specifically related to Drupal issues. I&#39;m sure there&#39;s more that&#39;s all that came to mind right now.</p> <p>You can always tweet at me if you disagree or have anything to add. Thanks.</p> Thu, 19 Dec 2013 00:00:00 +0000 http://www.darrenmothersele.com//blog/2013/12/19/drupal-common-mistakes http://www.darrenmothersele.com//blog/2013/12/19/drupal-common-mistakes