Overwriting the customize link with jQuery

To change page customize link popup to show in modal dialog you need jQuery and SimpleModal jQuery plugin.
Load SimpleModal library in page template header. If you are on APEX 3.x you need also load jQuery.

Then create HTML region to page zero without template.
Add to region source:

<script type="text/javascript">
$(function(){
 $("#customize a.tiny").attr({"href":"#"}).click(function(){
  htlmdbCustomizePage();
 });
});
function htlmdbCustomizePage(){
 var src="wwv_flow_customize.show?p_flow=&APP_ID.&p_page=&APP_PAGE_ID.&p_session=&APP_SESSION.&p_lang=&BROWSER_LANGUAGE.";
 $.modal('<iframe id="htmldbCustomize" src="' + src + '" height="400" width="600" frameborder="0">',{
  containerCss:{
   border:"0 none",
   background:"#FFF",
   padding:10,
   height:400,
   width:600
  },
  persist:true,
  overlayCss:{backgroundColor:"#606060"},
  onShow:function(){
   $("#htmldbCustomize").load(function(){
    $(this).contents().find("input[type='button']:first").removeAttr("onclick").click(function(){
     $.modal.close();
    });
   });
  },
  onClose:function(){
   location.reload();
  }
 });
}
</script>

NOTE! you might need change below jQuery selector depending on your theme:

$("#customize a.tiny").

Then enable users can customize which regions are shown. See documentation Enabling Users to Customize a Page.

When click Customize link, result should look like below.

See working example.